ElectricMotorcycleForum.com

  • May 03, 2024, 10:25:21 PM
  • Welcome, Guest
Please login or register.

Login with username, password and session length
Advanced search  

News:

Electric Motorcycle Forum is live!

Pages: [1] 2 3 ... 7

Author Topic: Log file parser utility  (Read 10967 times)

PhreaK

  • Full Member
  • ***
  • Posts: 143
    • View Profile
Log file parser utility
« on: September 17, 2015, 08:32:58 PM »

After a few cursory looks at the support logs that you can extract from the bikes with the Zero app + bluetooth connection, I thought I'd pop together a little utility to provide a nicer option than just viewing through a hex editor. It's still early days, however I've started to build out a map of the file and log entry format based on the logs from my bike, and those I've been able to pull from the forums so far.

I'll keep pushing updates to a GitHub repo, with the goal of building out some tooling to translate the *.bin files from the app into a nice, human readable form to assist your own diagnostics and curiosity. If I get the time, I may even end up rolling this into a free service you can email your logs to directly from the app and it will shoot you back readable versions... maybe.

If you have any suggestions or improvements, feel free to fork the repo and submit pull requests, or if you're not a coder, log an issue.

For those looking to help out, I'd love to get my hands on some more logs so I can figure out how the file structure clicks together. Feel free to drop me a PM or shoot them to contact@kimburgess.info. Also, if anyone has already dived into this in any detail, it'd be great to hear from you too.

P.S. I know there's some awesome tools out there already and you can hook into the TTL / RS-232 line on the OBD port (which btw, if anyone has concrete info on what it is so I don't need to chase down a meter / scope to test, I'd love to know). This is simply a way to get at least a little bit of info, without the need for any external hardware in lieu of official tools from Zero. Plus, it a fun nerd puzzle to put together.
« Last Edit: September 17, 2015, 08:41:47 PM by PhreaK »
Logged
2014 Zero SR

BrianTRice@gmail.com

  • Unofficial Zero Manual Editor
  • Hero Member
  • *****
  • Posts: 4014
  • Nerdy Adventurer
    • View Profile
    • Personal site
Re: Log file parser utility
« Reply #1 on: September 17, 2015, 09:29:13 PM »

Nice, thank you; especially for writing out the reverse-engineered format rules. If I find a reason to use it, I'll probably contribute.
Logged
Current: 2020 DSR, 2012 Suzuki V-Strom
Former: 2016 DSR, 2013 DS

Patrick Truchon

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Re: Log file parser utility
« Reply #2 on: September 18, 2015, 06:35:44 AM »

Cool tool!  I just emailed you a copy of my bin file along with the output that your script generated.  Hope that helps a bit...

Cheers!
Logged
Owned:  2013 Zero DS (now dead)
Test rode:  2014 Zero S, SR, 2015 Zero FX, 2016 DSR and FXS
http://ptruchon.pagekite.me/

PhreaK

  • Full Member
  • ***
  • Posts: 143
    • View Profile
Re: Log file parser utility
« Reply #3 on: September 18, 2015, 04:38:36 PM »

To everyone that's been sending me logs so far - thank you, you're awesome. After having a little ah-ha moment and realising all the numeric values are little endian and the timestamps are local rather than UTC, I've got the log entry message format sussed. In addition to the standard string messages, there's a whole array of entry types to now start decoding. This is going to be one huge, entertaining rabbit hole...
Logged
2014 Zero SR

BrianTRice@gmail.com

  • Unofficial Zero Manual Editor
  • Hero Member
  • *****
  • Posts: 4014
  • Nerdy Adventurer
    • View Profile
    • Personal site
Re: Log file parser utility
« Reply #4 on: September 18, 2015, 11:21:35 PM »

That's great! Yes, more data samples always challenge in a good way.
Logged
Current: 2020 DSR, 2012 Suzuki V-Strom
Former: 2016 DSR, 2013 DS

Justin Andrews

  • Hero Member
  • *****
  • Posts: 1032
    • View Profile
Re: Log file parser utility
« Reply #5 on: September 19, 2015, 01:41:39 AM »

What if I was to pull logs off my bike using my Data Reaper, and via the app at the same time?
Would that be useful?
Logged
Zero 2015 SR (+PT);
Yamaha Diversion 900

PhreaK

  • Full Member
  • ***
  • Posts: 143
    • View Profile
Re: Log file parser utility
« Reply #6 on: September 19, 2015, 04:53:14 AM »

Yes, that would be incredible. Then I can line up some more of those message types.


Sent from my iPhone using Tapatalk
Logged
2014 Zero SR

Christian J

  • Newbie
  • *
  • Posts: 6
    • View Profile
Re: Log file parser utility
« Reply #7 on: September 19, 2015, 11:07:38 AM »

Hi Kim,

I added some of my logs to this post.

I was unable to parse them (used python 3.5). Maybe you can look into this.

Quote
>python zero_log_parser.py 538SM5Z36ECG03915_MBB_2014-10-13.bin
Traceback (most recent call last):
  File "zero_log_parser.py", line 81, in <module>
    parse_log(log_file, output_file)
  File "zero_log_parser.py", line 63, in parse_log
    if ord(b) == ENTRY_DELIMITER:
TypeError: ord() expected string of length 1, but int found

Thanks for your effort. Hopefully one day we can set this up as a web service - that would be perfect.
Logged

PhreaK

  • Full Member
  • ***
  • Posts: 143
    • View Profile
Re: Log file parser utility
« Reply #8 on: September 19, 2015, 08:43:41 PM »

Thanks Christian! Ignore that python script for the moment. That version was just a quick stub and some straight bin -> ascii / utf-8 translation to make seeing what was there simpler. I'm focussing on documenting as much of the file structure as possible first, then I'll roll that into tooling.
Logged
2014 Zero SR

Justin Andrews

  • Hero Member
  • *****
  • Posts: 1032
    • View Profile
Re: Log file parser utility
« Reply #9 on: September 20, 2015, 09:49:08 PM »

Phreak, here are my logs and the Data Reaper log (in two parts)
Logged
Zero 2015 SR (+PT);
Yamaha Diversion 900

Justin Andrews

  • Hero Member
  • *****
  • Posts: 1032
    • View Profile
Re: Log file parser utility
« Reply #10 on: September 20, 2015, 09:49:40 PM »

Part 2 of the Data Reaper log from the MBB
Logged
Zero 2015 SR (+PT);
Yamaha Diversion 900

PhreaK

  • Full Member
  • ***
  • Posts: 143
    • View Profile
Re: Log file parser utility
« Reply #11 on: October 03, 2015, 06:59:01 PM »

For those that have run this already - should be a little more useful info in those decoded logs for you now. The script is probably still buggy as hell, and may break a lot, but I'll keep chipping away as I get time.

I was running through some ideas as to how this could be rolled into a web service, mainly for my own entertainment (I'm not a web dev, but sort of interested in learning a little bit of that world). Out of interest, what would be peoples reaction to the following:
  • Logs may be uploaded, or emailed, allowing you to just extract with your phone and change the address from zero support to the service address.
  • If emailed, decoded logs will be emailed back. If uploaded the results are shown / available for download.
  • System keeps track of firmware revs from submitted logs. This way it can also provide an alert service for new firmware release availability as (from what I can tell) there's no mechanism for this currently without dragging your bike to a dealer.
  • Maybe some form of analytics on common error log entries?
Logged
2014 Zero SR

Patrick Truchon

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Re: Log file parser utility
« Reply #12 on: October 03, 2015, 11:12:58 PM »

Very nice work!  I've been parsing the same file with your different versions since September 17th, and every time, more gets revealed.  Pretty cool!

As for the web service, I don't think I would personally use it but others might.  I know I'll be keeping your script in my collection very preciously...

Looking forward to more updates.  Keep up the excellent work!
Logged
Owned:  2013 Zero DS (now dead)
Test rode:  2014 Zero S, SR, 2015 Zero FX, 2016 DSR and FXS
http://ptruchon.pagekite.me/

pablogomez

  • Newbie
  • *
  • Posts: 2
    • View Profile
Re: Log file parser utility
« Reply #13 on: October 12, 2015, 06:52:00 PM »

Hi I send to you the logs, Another question is how can I know the version I'm in?
Logged

Patrick Truchon

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Re: Log file parser utility
« Reply #14 on: October 13, 2015, 08:00:15 AM »

Hi I send to you the logs, Another question is how can I know the version I'm in?

Hello pablogomez, if you download his script and run it on your file, you'll get a text file with more than a thousand lines.  Yours starts with this:

Code: [Select]
Serial number      2014_mbb_0386e1_00235
VIN                538SD5Z28ECB03553
Firmware rev.      36
Board rev.         1

Hope this helps.

I'm not sure if Kim needs anymore stand alone bin files.  I think what he's after now is files from people that have the Diginow Data Reaper Dongle so he can compare the input and the output.
Logged
Owned:  2013 Zero DS (now dead)
Test rode:  2014 Zero S, SR, 2015 Zero FX, 2016 DSR and FXS
http://ptruchon.pagekite.me/
Pages: [1] 2 3 ... 7