So, I have been having intermittent and infrequent CAN issues (P1797, U2500, reduced performance mode, often stalls), but I have run out of things to clean, check and/or resolder. The ODB II codes aren't particularly helpful, but I fortunately had a CAN analyzer on the car when it happened recently. Bottom line, the ECM token disappears with all of the ECM CAN IDs. This seems to indicate that it is not CAN connectivity since all of the other messages (TCM, ABS, INST) are still being transmitted, through the wires that go through the ECM. Turn the car off and on, then it's better, as always.

So, that led me to start decoding all of the CAN messages so I could track them dynamically and gain insight before just blindly replacing the ECM. These are probably relevant for at least 2001/2002. Since SAE J1939 was young when my car was built (1998 revision), I wasn't clear how easy it would be to decode. However, the order of messages in the Electrical Guide appears to be the order of the messages by byte/bit, which is helpful. There are only 13 normal IDs with 71 parameters (SPNs) which is also helpful. I think there may some variance from J1939-71 (1998). For example, the recommended SLOTs (scaling, limit, offset and transfer function) appear to be sometimes followed, and sometimes not.

In any case, attached are the 9 instrument (INST) messages with ID and byte/bit locations. They have been checked against car state, where possible. For example, I wasn't going to induce a low oil pressure state just to check the relevant message. Only 62ish more to go...

 

Hey! Thanks for doing this!

I might end up playing with the can bus later on to develop things with microcontrollers.

You are doing the hard work for the rest of us that will eventually dig in and use this.

 

Has been interesting....

So, the messages do appear to be in the same order as the XK8 2001 electrical manual. Some messages are quite clear, some never deviate from zero given my car and my data collected. I initially started with the relevant standard, SAE J1939-71, versions from 1998 and 2011 were the ones I had access to. 1998 seemed a plausible basis. I think using these slowed my progress. Perhaps the original 1996 version would be better, but, for instance, the transforms between bytes and measured values (liters, kph, kilometers, percentages, etc) often do not follow the SLOT recommendations from the standard. So, some of the transforms are still unclear. The number of bytes/bits for a given defined message also often deviates from the standard, etc.

Would have been faster to start with the easy phenotypes (e.g. brake on, parking brake set, wheel speed, set speed, target speed, etc) and fill from there with few assumptions. However, if I had assumed that the order was random, not as in the electrical manual, that would have made things much longer. Yin, yang.

In any case, attached is my current thoughts on everything from the four modules (ECM, TCM, INST, ABS) save the diagnostic queries and responses. In yellow are the ones that are more uncertain, at times completely. I think this message set applies to 2001-2002, uncertain how it applies beyond this.

As always, no warranty is implied, no suitability for any purpose is stated or implied, no deposit, no return, etc, but please feel free to amend, append, transcend...

 

Please can you share how you collected this data? What hardware/software you used? Apologies if you have already posted this.

 

I don't know exactly what CAN sniffer he is using right now, but the technology is going to be called a CAN bus analyzer, sniffer, or spoofer depending on the exact functionalities available to the hardware/software combination.

Essentially it is a small computer that intercepts the raw packets of data being sent on the CAN, either by being spliced directly into the HI LOW twisted pair, or by plugging into a premade interface like the OBD2 diagnostic port.

The packets of data being sent on the network are going to conform to a specific ISO standard, which standard is quite the question though as crbass is discovering. More than likely Jaguar based their communication protocol off of one of the most popular standards at the time, and then *******ized it when the need arose.

crbass is intercepting all of the packets being sent on the network, and then using process of elimination to translate the packets from hexadecimal or some other format, into human readable naming conventions. The data being sent associated with the message at hand may also be a range of values, a digital on/off value, or seemingly random values. So he is using process of elimination to attempt to figure out what values do what when being sent with a particular message.

I believe he is cross referencing the CAN message list that is provided in one of the FAQ posts to make the job easier to translate some of the messages.


As for hardware that you can buy premade to intercept these packets? You can google CAN Analyzer or CAN sniffer to find tons of products that exist to do this purpose ( most automotive related ). You also can create your own with an microprocessor kit like atmel or arduino or raspberry pi with an appropriate can bus shield.

Here is a product that claims to be made for doing exactly what crbass is doing. https://www.csselectronics.com/scree...rk/language/en

As for the software, there are tons of network inspection softwares that exist. Most of the automotive tools will have their own program or app made specifically for their piece of hardware they are selling.

The most popular free network inspection tool is called WireShark and the product in the link above also goes into a bit of detail of how you can use wireshark with their tool to sniff the network and spoof some messages.


In case you already knew most of this information, please don't think I am trying to talk down to you. I just imagine not everyone on the forum has this baseline of information as to what crbass is doing for us all and maybe my long post can help someone enter the discussion where they were uncomfortable to talk beforehand.

 

@Mad Hatter thanks - I know all that, I was rather hoping for some specifics as to exactly what hardware/software combination the OP is using that works on our cars. It would be nice to know exactly how to go about doing the same.

 

FWIW, the super cheap ELM327 devices can read all car protocols, including CAN. The Bluetooth interface to your mobile device will likely not handle the entire flow of all messages, but the device has handy filters to make things a lot more manageable. The ELM327 datasheet is a great read for someone interested in getting a good primer on car data networks and diagnostics. One of the complication/frustration is that the base technologies are "sponsored" by big standards organizations (ISO, SAE) that only release the underlying documents for a fee, so getting into the details of CAN message formats and conventions is more difficult. On top of that, even as manufacturers "share" the proprietary data network message documentation, they only do so with an organization asking for rather hefty (and recurring) access fees.

 

He may already be using an elm327 as they can grab all messages if asked.

 

Thanks for asking, dibbit. No, but I should have included this.

As noted above, there are many options available at this point. I happen to have a Picoscope 5444 for doing science which has various decoders, one of which is CAN. This is not the low cost solution here (about $2800), the ELM 327 fmertz discussed above looks like an interesting option. The only drawback I can think of would be the issue mentioned by fmertz, the frame rate. The XK8 is transmitting at 500 kbaud (500,000 state changes (or bits) per second). So, depending on how the buffers/transmit in the ELM work, it may or may not be able to keep up, but that is fairly easy to check. Theoretical 3.0 bluetooth speeds go to about 25 Mhz, but in actual practice, this has never been achieved by me, personally. Or anything close to it. I am also using a cheap OBDII breakout board since the pico is just a cheapish but fancy multichannel oscilloscope.


Early on, I had some larger error rates (about 1/5 of messages) on the CAN sniffing because of bad assumptions on hysteresis on my part. I suspect with the cheaper devices, this be done for you without having to know how the signal should behave. This isn't happening any more, I get data that fails CRC (cyclic redundancy check) about 1/1000 messages now. Since the bus signal is visibly noisy, that seems okay. I could probably improve this, but see no need. Also, the actual noise on the underlying signal was what I thought was my problem in the first place, it wasn't. Turns out my ECM occasionally but rarely stops sending any signal, including the ECM token, to the bus for a period of seconds, have finally caught this when it happened, twice. That does not make the car happy (reduced performance, ABS complains (U2500), TCM complains (P1797)), but it surprisingly keeps running anyway, most of the time, since the whole bus seems unimpaired.

I also have a Foxwell OBDII scanner (NT510) with the JLR software. That has some not insubstantial quirks, but I didn't buy it for this. For example, it finds no codes when the modules are scanned individually, but finds them when it does autoscan on all of them. In any case, I determined the call and response diagnostic IDs from the Foxwell. Those are not transmitted unless something that is connected to the port first transmits the call ID with a message.

A bit on the SAE standard, the first version was promulgated in 1996. The first 'real version' (my opinion) was in 1998. It has been updated many times since. It is clear that early engineering on the XK8 could not have had the 1998 final, but very often, the teams have prefinal versions that is so close to the final that there is no difference. For example, a recent ISO standard I wrote large bits of was essentially final for 7 years before it was passed. Again, a bit surprised that many of the defined messages in the standard are not as defined in the car.

But, there is no obligation for the manufacturer to use such a standard, except that the hardware people are probably using it.... and CARB (California emissions people) was demanding OBDII things as a prerequisite for selling your car in California.

And yes, I used Nerobergstr's original identification of the 13 IDs for the 4 modules which was correct, crossreferenced with the XK8 2001 Range Electrical Manual and just recorded while I did various things with the car. E.g. cruise at 60, cruise at 45, cruise at 35, cruise at 15 (this doesn't work, by the way, in case you ever wanted to cruise at 15 mph...), skid with ABS on, skid with ABS off, punch the accelerator pedal going uphill, change gears, etc. The big, full byte or multibyte, things were relatively easy to find (e.g. coolant temperature, wheel speed, engine rpm) within the messsages, which are up to 8 bytes. Within bytes, it was occasionally hard to determine where one ended. Some things are still mystifying...

 

Have had a tiny amount of time to continue figuring out the CAN messages for the XK8 (MY2001-2002, but I expect some measure of commonality before and after).

I now have a CL2000 CAN bus logger (https://www.csselectronics.com/scree...00/language/en) to play with which is much less cumbersome than my earlier equipment. I'm just leaving this on the car at all times, now. It also has the facility to inject messages to the bus, if desired.

Someday, perhaps, I plan on getting around to determining the response messages from IDS queries to the modules and the code reader queries to the modules. This may include messages from the SCP bus.

But, until then, another revision of the CAN messages below:

 

I did end up getting C1000 from them as well.
Didn't end up ordering a cable through them and had to cut the one apart that I ordered from ebay and change the pinouts.
After that it worked perfectly and was completely plug n play.


I would love to hear what script you were using to format the data log dumps i was sending you, it was immensely more readable after you formatted it.

 

Funny that cable thing. I did order one from CSS, but it only fit into the car with a bit of force. That being unacceptable, like you, I rewired one of these


If anyone else is interested, pinouts are in the CSS manual, and/or I would be happy to share which wires need to be crossed.

It would have been better for me if CSS had a OBDII plug instead of the DB9, but the cable adapter allows one to put the module in the back seat floor, which never sees any sunlight in my car. Like you, it worked well right out of the box for me. I did not buy their outrageously priced 32 GB SD card, but replaced the 8 GB one with a cheap 32 GB card when it arrived. Also plug and play. Did not check to see if it would work with larger capacity cards since 32 GB is a month or more of data.

Unfortunately, the 'script' is simply excel with your data reformatted to my original template. I'm very slowly working on something better since now I have much more data and it clogs excel.

 

I am actually a programmer by trade. Could write something for you quite quickly if we were to get together in a conversation about what exactly needed to be done.

I feel like you have a bit of information about the message formatting and hex encoding that I am lacking at the moment.

 

I came across this thread a few weeks ago, and I am very impressed at the amount of work that went into decoding the messages!

I checked into the operation of my 2002 XKR, to compare with XK8 results, using a PC and Arduino + canbus module, doing real time dump of about 40 secs worth at a time (could do more but files start to get big) - and no surprise that the formats are very similar
- ABS outputs at about 7.75 ms periodicity - maybe this is within the error range of the 8 ms estimate on the XK8. Also the heartbeat is 260 ms unless I miscomputed.
- TCM outputs seem a little different (since it is the Mercedes box not the ZF)
- the 0x0C8 message outputs at 20 ms periodicity, and 0x071 heartbeat also at 20 ms, as far as I can tell

All in all the messages are pretty much identical.

I then did a quick bit of logging of IDS requests, per idea in post #10, to read custom PIDs, and got the following:

Single handshake exchange as follows:

SID 0x7e8 (Diagnostic to ECU), 8 byte message (in hex) 01 10 00 00 00 00 00 00
SID 0x7ec (ECU to Diagnostic), 8 byte message 06 7f 10 00 00 00 00 00

Not sure what the purpose of this is, other than a basic handshake to valid the connectivity - anyone have any ideas?
Format suggests single byte request, service 0x10, and 6 byte response of ??

Then continuous reporting of the target PID (timing is quite scattered, will need more analysis to get accurate estimates)
Handshake every ~60ms
Example, engine rpm (with engine off, i.e. value of zero)
SID 0x7e8 (Diagnostic to ECU), 8 byte message (in hex) 03 22 00 0c 00 00 00 00
SID 0x7ec (ECU to Diagnostic), 8 byte message 05 62 00 0c 00 00 00 00

This seems to match the custom PID request/response format, where for request
byte 0 - 3 byte request
byte 1 - 22 - service ID for enhanced diagnostic data
byte 2, 3 - PID value

and for response:
byte 0 - number of following bytes
byte 1 - service ID + 0x40
byte 2, 3 - PID value
bytes 4-7 (depending on byte 0) - data for PID

Since PID 00 0c is well known PID for rpm, things seem to line up, and value was 0 which is what it was with the engine off. I'll have to do more testing under various engine conditions.

So armed with this, I tried a couple more PIDs

O2 sensor upstream (wideband) for banks 1 and 2
- PIDs seem to be 00 34 and 00 38 (these are standard PIDs so I need to recheck I did not select downstream by mistake, although I'm fairly sure I selected upstream)
- data is reported as 4 bytes, I am wondering whether format is per standard O2 current sensor PIDs - 34 and 38, i.e. two parts
- Equivalence ratio as ((A*256)+B)/32768
- current in mA as ((C*256)+D)/256
- with the engine off, IDS reported -60 mA and PID report was 80 0 41 03, which would give -63 mA I think, so not sure if this is a good interpretation, although sometimes IDS rounds the results unless the reporting range is set to a smaller window of values.

TCM transmission oil temp
This was an exchange with the TCM, so addresses are different although single message handshake, and then continuous reporting seemed to follow the same format

Single handshake
SID 0x7e9 (Diagnostic to TCM), 8 byte message (in hex) 01 10 00 00 00 00 00 00
SID 0x7ed (TCM to Diagnostic), 8 byte message 06 7f 10 00 00 00 00 00

Continuous reporting (engine off, IDS reported fluid temp of 20 degrees C, range -50 to 200 degrees
SID 0x7e9 (Diagnostic to TCM), 8 byte message (in hex) 03 22 11 bd 00 00 00 00
SID 0x7ed (TCM to Diagnostic), 8 byte message 04 62 11 bd 46 00 00 00

46 hex is 70 decimal, -50 + 70 = 20 degrees, so looks good.

So next step would be to check these out with a standard reader (assuming it targets the canbus rather than the other buses) but I seem to have lost mine for now, so if someone wants to try it out and report back, that would be interesting. If someone has a favorite parameter to go searching for, I will try and find time to capture the PID.
Note these codes, especially trans fluid temp, likely won't work on XK8 since the TCM is different. Whilst I have an XK8, I don't want to mess with the wiring, and I don't have a spare OBD2 plug lying around, so can't log the XK8 for now.

Bottom line is that with a bit more validation, it looks like a reasonable method to get custom PIDs for modules that are on the 500 kbaud canbus (as opposed to the other two slower buses - SCP and Serial). Of course if that weird handshake is required as a precursor, that may mess things up - or it may be my limited understanding of the protocols involved....

Potential requests with service ID 0x22 - enhanced diagnostic data
O2 Upstream Bank 1 PID - 00 34
O2 Upstream Bank 2 PID - 00 38
Transmission oil temp PID - 11 BD

Maybe someone has done the hard work on these already, but since in my searches on the forum I could not find any info, I thought I would post my findings in case it is useful.

 

I have also been fiddling with some can bus message decoding recently. Did the actual logging November and December last year but still havent figured it all out. Engine oil temp I cannot figure out from my logs. My car is a Xkr 2000. Engine oil temp should be found within the 0x05C or similar hex pid according to J1939 standards, but those pids never occur in my logs. Anyone found out what pid to display the engine oil temp?

I can see that Crbass points the engine oil temp to 0x044 but he hasn't confirmed it. If the Engine oil temp has an offset of -40 just like the Engine coolant temp that can be found in bit 1 of the 0x044, then there is no value for Engine oil temp in my log

 

Try sending to the TCM these (hex) and see if any looks plausible
22 11 BD
22 16 74
22 10 0C
22 10 0D
22 10 0E
22 10 0F
22 1E 69

 

Quick follow up on the custom PIDs

The single request message 01 10, i.e. a request for mode 0x10 (which is not one of the ten standardized modes) seems, as suspected, to be an 'enable' message to get the module to respond to custom PID requests - without this no response is received. After issuing this, the responses do occur.
This was news to me, maybe not to others...

I checked this with Torque Pro to try it out and it worked, although it's messy because on my '02 XKR Torque defaults to using OSI 9141-2. So i decided to force it to use the canbus, setup the custom addresses (since the 'standard' interval of 8 between request and response is not used, rather the interval is only 4, i,e, ECU request is 7e8 and response is 7ec (not what would be a more standard 7e0 and 7e8), and issue the 'enable diagnostic' command of 0x0110. If you are familiar with ELM327 commands and Torque, you can probably figure it out in a nicer way than I did, but I created a custom PID for TCM temp as follow
- PID 2211bd
- min value -50, max value 200 (in celcius, if you default to fahrenheit then adjust the formula and min/max)
- Equation: A - 50
- OBD header to use: 7e9
- Diagnostic start command: atsp6\natcra7ed\natcaf0\n0110\natcaf1\n
[set to mode 6 which is canbus 11 bit 500 kbaud, set receive to 7ed, disable auto formatting to allow sending of the 'mode 0x10 message, send 0110, re-enable auto formatting]
- Diagnostic stop command: atsp3\n
[set mode back to OSI 9141-2]

- I also tried to simultaneously log the upstream O2 sensors with similar scripts (addressing ECU not TCM) and it kind of worked, but only intermittently, so I probably need to refine my Torque skills....or I am missing something else.....
Anyway, I don't want to turn this thread into a custom PID discussion, I was just trying to validate the principles, and I think I'm done there.

 

I think this is great, in general. My long ago plan was to map the CAN diagnostic requests/responses (over the course of the next geological age) and the SCP messages (call/response). What you are doing starts it and having the compilation of the PIDs will likely be helpful in the future. There is probably much commonality among the 1997-2002 at least, and probably through 2006.

One thing to note is that the 'standard' CAN things hadn't been set when the XK8 was designed (1996), to say nothing of the evolution of the standard from 1997. So, you may note that almost nothing follows the SAE CAN standard, exactly, when it follows it at all. So, by your example, 7E8 is the ECM call, but 7EC is the response, not the current recommended 7E8/7E0.

So, I have a Foxwell scanner, and I did pass by the car last night and set up to pick off the interactions between the car and the Foxwell scanner, at least for the diagnostic case where there are no codes (no custom PIDs yet, just the basics). Satisfyingly and amazingly, no codes right now...
Results are below. Many things I don't understand about this, but at least the call and response IDs are what I expected. Sometime, I'll make codes happen and parse out that structure, and proceed down the available standard PIDs, probably with IDS since the Foxwell has issues with most of the things it says it can do.


Listening in on the CAN interactions between the Foxwell and the 2001 XK8

 

the plot thickens...a couple of observations

I thought it possibly sneaky that the apparent 'enable diagnostic mode' request (i.e. 0x 01 10), if that is really what it does (but IDS must send it for a reason), receives a failure code rather than an acknowledgement, even though it potentially seems essential to enable responses i.e. the 0x 06 7f 10 response [6 bytes - error - requested service 10 -, ...<no response failure code provided in remaining bytes... >. At least my basic understanding is that this is an error response (although I don't have a copy of ISO 15031-x and as you say it was possibly not followed even if it was available in '96)
- second line of your table - ABS response, was the response really '06 7f 20', and not '06 7f 10' to a '01 10' request, or a typo?

INST responded to a 22 service request without the 'diagnostic enable' command 01 10......so no enable needed for that module?

TCM responded to a 20 service request with the 7f 20 error response, and your scanner did not request a 10 service....
- so there is a possibly 20 service........whatever that is....if the 7f error response is as misleading as with the potential 10 service
- and possibly you got a service 22 response before issuing any other 'enable' command (01 10 or 01 20...), which would be perplexing because I needed to issue 01 10 to get the TCM to respond to my 22 service request (11 bd PID)

PID 200 using service 22 gets different 1 byte responses from INST and TCM (01 and 00) - so there's a PID 200.....

[One other thing in the back of my mind relative to operational data on the canbus - 684SG advanced electrical course has a statement relating to canbus "Typically multiple vehicle operation data frames are transmitted because of the amount of data to be sent, and the limited capacity of each frame." - so possibly multi-frames are used on some messages (like ISO 15765-2?), although I'm not sure that is true....maybe only applies to later models....I don't think you saw any evidence of that in your analysis?]

 

Yes, interesting. No typos in the data, the data is right out of the data acquisition, mysterious. And I agree, 7F is probably an error response. On the TCM response, there is a possibility the data acquisition missed the call (because of an error), I just haven't had time to run it again to see what happens.

And no, in this extremely limited test, I see no evidence of multi-frames, but there's essentially no data here. Will need to think how I give one of the modules enough codes it would need several frames.

I'll see if I can scare up ISO15031-5:2006 (version 1) and see if that's helpful. Probably not, but no harm in looking. I do have J1979-1997 and 2001 somewhere, that might be helpful here.