Models vs Checkers/Scoreboards

dave_whipp · Posted: Mon Oct 18, 2004 8:55 pm **Post subject:**

There are a number of ways to tackle this question, but I think that, for any non-trivial high level behavior, a model is easier to both verify and validate.

As a "simple" example, consider the assertion "this machine has a behavior consistant with the X86 instruction set architecture". The standard implementation of this assertion is to create an ISA model, and create a smart-diff script to compare transaction/state information that are dumped from rtl and model.

The "partial function" point is relevant, but is not exclusive to the assertion approach. The context of the quoted text was discussed the "rtl synchronization" approach, where don't care values are fed back from rtl into the predictor.

Imagine we are doing a simple square-root function, where we have a desired accuracy of +- 1e-6, but the rtl calculates it to a greate accuracy to avoid loss of accuracy in future calculations. This means that the value is a combination of "do care" and "don't care". We can implement this in a synchronized model, by used an assertion of the value we get (assert abs(rtl_value ^2 - predicted_value) < 1e-6), and having validated it, we can use the rtl's value to keep the predictor synchronised with the rtl.

A similar approach can be used anywhere where a partial prediction is needed: add an assertion to validate the value, then use the actual value.

Using a model does not prevent us from using assertions, etc. It just one tool in the toolkit.

The idea of modularity is nice, but features aren't really as orthogonal as we might wish. If the purpose of a test is to draw a teapot, then that test is exercising thousands of individual features, and probably checking 10s of millions of instances of those features. Even if you have assertions for each individual feature, you still want to know if, when they all interact, the picture looks like a teapot. If you need modularity, then to create the model as modular software.

alexg · Senior Joined: Jan 07, 2004 Posts: 586 Location: Ottawa

dave_whipp · Posted: Tue Oct 19, 2004 2:04 am **Post subject:**

alexg · Senior Joined: Jan 07, 2004 Posts: 586 Location: Ottawa

dave_whipp · Posted: Tue Oct 19, 2004 11:33 am **Post subject:**

Lets say that I accept your analogy of "putting a new engine oin an old car" as a proxy for a model. Its not perfect, bt it is adequate to demonstrate why models are important during verification.

Your use of the model is simply to compare maximum speed -- which, as you say, is better done by direct measurement, and so is a waste of a good model.

A better way to use it is to drive both cars round a track for a few hours, with different fuel loads, tires, etc. We continuously record as much telemetry as possible fromthe two vehicles -- but to keep things simple, lets say we just have the speed measured every second over a 12 hour test.

What do we do now? We plot the data from both cars and overlay the graphs, WE then look to see how they differ. As a result of this comparison, we notice that when changing to 4th gear on a left hand bend with light fuel load, the acceleration of the new car has a slightly differnt profile compared to the model (its still within spec -- but it catches our eye because it wiggles relative to the model). We decide to investigate this difference, and ultimately find a reliability issue that would have reduce the MTBF for the gear box by 6 months.

You don't use a model to measure a single fact. You use it to make a prediction of complex behavior (over time), and then detect divergence from that prediction without needing to predict where that divergence might be.

alexg · Senior Joined: Jan 07, 2004 Posts: 586 Location: Ottawa

Dave,
You are assuming that your reference car (model) is free of bugs. I am assuming that the fact that car is simply different (model is written using another language) doesn't make it perfect by definition. Any models, especially complex ones, require verification too, which means - testbench, test, checker, coverage points etc. Any modification of complex model requires regression tests on the model.

For checkers, due to their modular srtucture and ability to implement any uncomplete functionality checks, this process is significantly easier.

Regards,
Alexander Gnusin

dave_whipp · Posted: Tue Oct 19, 2004 3:17 pm **Post subject:**

I don't know where you came up with the idea that I don't think that models (or tests, or assertions) contain bugs. My experience is that the RTL is usually the last place to look: we sometimes joke that the purpose of the RTL is to find bugs in the model. When you are using a model, the thing you look for is a difference between the prediction and the observed behavior. Once you find a difference, you then dig in to localise the difference (model, rtl, diff-script, nfs-cache corruption, etc.), and then work out the root cause.

One of the problems I find with complex assertions is that I don't know how to test them. False-fail is easy to detect, but how do you test that an assertion is not giving you a false-pass? I've seem methodologies where tools insert random bugs into a design to make sure they see every assertion fail, but those tools are not mainstream.

I don't understand your final paragraph" "for checkers, due to their modular nature...". In what way is a system of checkers more modular than a properly written model? And "... ability to implement any uncompelete functionality checks": my first post in this thread explained how to implement incomplete function modeling.

alexg · Senior Joined: Jan 07, 2004 Posts: 586 Location: Ottawa

dave_whipp · Posted: Tue Oct 19, 2004 10:51 pm **Post subject:**

"I made wrong conclusion from your previous post ... you compared ... to reference"

The act of comparing tells you there's a difference. It doesn't tell you which is right (if either). One danger of simple model Vs rtl comparison is the possibility of common-mode errors. As a rule of thum, you want 3 descriptions of any behavior. So, for example, you might use a self-checking test as the third description.

I can see that I was reading too much into your use of the word "assertion". Checkers as you descibe them are something else entirely.

The methodology you describe is what I think of as the "Testbuilder" methodology, because the first place I saw it really promoted in public was at an Testbuilder (Cadence) seminar about 5 years ago. A company named Azanda presented as a "Success Story": they taped out a switch fabric using a methodology that sounds identical to what you describe. There are probably some papers on the web somewhere

I felt at the time, and continue to believe, that the methodology doesn't scale to complex systems (switch fabrics are somewhat regular, and ahve moderately simple end-to-end behavior). If end-to-end behavior is more than the sum of its parts, then the highest layer of checkers has to span the whole system. I'm not sure that it has to be strictly hierarchical, though Testbuilder used C++ in a non-object-oriented way, so I tend to assume that it is hierarchical.

The thing is, if you push too far in the direction of multi-layer non-hierarchical web of intercommunicating checkers, then what you end up with is a transaction model of the system: created bottomn up. And if you don't push that far, then you'll have holes in your coverage.

A actually agree with your final paragraph: verification of a complex system is much more complex than simply setting up a set of protocol checkers on the interfaces between [sub] units. As Einstein said: "things should be made as simple as possible, but no simpler".

alexg · Senior Joined: Jan 07, 2004 Posts: 586 Location: Ottawa

dave_whipp · Posted: Thu Oct 21, 2004 12:19 am **Post subject:**

dave_whipp · Posted: Thu Oct 21, 2004 1:04 am **Post subject:**

I need to qualify my previous post. When I said it would be "stupid" to create a model by copying an implementation, that was a bad choice of words. Although it was aimed at the act, not the person, I apologize if any offence is felt.

In addition to it being a bad word choice, I must also nuance the underlying sentiment. As part of a well thought out strategy, it might make sense if it enabled a bottom-up validation flow where a fast model is able to run through a wider range of scenarios that a slow RTL simulation.

alexg · Senior Joined: Jan 07, 2004 Posts: 586 Location: Ottawa

jmcneal · Posted: Thu Oct 21, 2004 10:59 am **Post subject:**

Alex, Dave,

For any complex property/function that needs to be verified on a large chip, isn't it often necessary to use a model in order to perform checking?

For example how would you just check correct routing/switching without making use of a model of the routing/switching algorithm? To verify the functionality you have to do some sort of lookup on a table to determine where a packet should emerge. Would you classify a piece of code that does a destination lookup in a routing table to predict where a packet should emerge as a checker or a model?

j