Highest Performance from your HVL Tools with Fewer Licenses.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 3.15, Item 5.0)

From: Tom West

The Problem:

1. High Level Verification Language tools throttle performance of
simulations when linked with RTL simulators through PLI.

2. The price of the HVL tools is prohibitively expensive to use in
regression mode if a one for one ratio of HVL to RTL simulation
licenses are required as in the case with PLI linked environments.

Some Assumptions:

* Lets assume we have an environment that is unbounded by hardware
processing horsepower. This is easy to achieve especially in the
case where environments utilize banks of Linux OS on Intel
processors.

* Lets assume we are unbounded by the quantity of RTL simulation
licenses. This is easy to achieve as RTL simulation is now a
commodity item.

* Lets assume we can generate stimulus at a rate 5 times faster than
the RTL simulation tools can consume it (True for my experiment).
This is easy to test for and is easily achieved in many environments
even with complex stimulus generation schemes. A corollary to this
assumption is that in a PLI linked environment, one could consider
that the HVL tool is idle 80% of the time. Is this an efficient use
of the premium priced HVL tool?

A Solution:

* Lets de-link the simulation tool from the RTL environment. Does
this sound like a step backwards in progress? Yes, There allot of
issues and features to give up, but lets focus on the up side.

* What if we could now use fewer HVL licenses than RTL simulator
licenses? What if we could not only save license usage by using
fewer simulator licenses, but gain considerable performance at the
same time?

My experiments utilized 5 HVL licenses (Specman), 25 RTL simulation
licenses (NC-Verilog) and 30 processors. The 5 HVL licenses are
working on 5 processors generating stimulus and expected results to
ASCII files. Multiple stimulus and expected result files get created
by each processor, i.e. The environment creates many smaller tests
rather than fewer longer tests. This ASCII file I/O approach gives me
the advantage of simulation logging while creating stimulus and
results.

A script kicks off all 5 HVL "generators" and monitors the completion
of the expected result and stimulus file creation. As the files are
generated, the same script kicks off RTL simulations on the other 25
processors. Each RTL simulation job would read the files, cycle the
data into the RTL and produce simulation results to an output file.

The script then proceeds to do a cycle for cycle comparison on the
expected results with the RTL results. If the comparison was correct
the files would be deleted, including the waveform files to save disk
space.

The Major Drawbacks:

1. The reactive nature of the test bench environment is lost. The
environment will have to be carefully planned to avoid the use of
feedback from HDL simulator.

2. Using the HVL tool for RTL functional coverage is not possible as
it requires constant links through PLI. However we can write our
own Verilog Models to perform basic RTL coverage, and coverage of
the stimulus from the generation engine is still available in the
HVL tool.

Summary:

The results I produced were crude but effective. The Ratio of 5 HVL
licenses and processors to 25 RTL simulators and processors achieved
almost perfect harmony of file creation to file consumption. With
this approach I used 25 fewer HVL licenses, while increasing my
simulation performance slightly over the PLI linked approach.

Another way to look at it is if I only had the 5 HVL licenses, I
boosted my overall performance by slightly over 5x the PLI linked
approach.

I hope to refine this further in the future.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 4.1, Item 5.0)

From: Chris Spear

> 1. High Level Verification Language tools throttle performance of
> simulations when linked with RTL simulators through PLI.

Vera and VCS simulate with very little overhead as they communicate
using Direct Kernel Access rather than using the PLI. The speedup
depends on design style and interface activity, but these run
typically 50% faster than Vera + NC-Verilog, and speedups of 3x have
been seen.

Tom was discussing Specman's simulator interface. I will be happy to
help him compare performance of this point. if he wishes.

> 2. The price of the HVL tools is prohibitively expensive to use in
> regression mode if a one for one ratio of HVL to RTL simulation
> licenses are required as in the case with PLI linked environments.

The alternative is to use Vera which has both a full license
(Compiler, Debugger, Run keys) and a less expensive regression license
(run key only). The latter is priced about 75% cheaper so customers
can set up simulation farms. Tool performance and cost are two of the
metrics used when comparing tools.

Of course if price is very important, Cadence's TestBuilder is another
possibility. It is more of a class library than Hardware Verification
Language.

In the end Tom has a very difficult job, balancing the number of bugs
found vs. time to market vs. a budget for EDA tools. Synopsys
Application Consultants are always ready to help our customers get the
full advantage of their tools.

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 4.1, Item 5.1)

From: David Ljung Madison

> 1. The reactive nature of the test bench environment is lost. The
> environment will have to be carefully planned to avoid the use of
> feedback from HDL simulator.

It's not clear to me why you'd use an HVL if it wasn't for this
reactive nature. What does an HVL buy you as merely a generator as
compared to any decent scripting language?

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 4.1, Item 5.2)

From: Zeev Kirshenbaum

I think this article is missing the point. The point is that
verification engineers need to verify that their design is functioning
correctly, and that its complete functionality was verified. The
question then is, does the proposed solution provide that? I think it
does NOT.

The approach of generating inputs in ASCII files, and then driving
them during simulation is not at all new. Some of Verisity's more
advanced customers have been using this approach in the past, before
they adopted Specman Elite. They chose to abandon that approach
because it has several key drawbacks, some of which Tom West
recognizes. Here is a partial list:

1) Lack of on-the-fly generation

On-the-fly generation is key for efficient constrained-random
verification environments. On one hand, you want to randomize as
much as you can to explore and find bugs you "can't think of". On
the other hand, some scenarios are so rare, that it would take too
much time for just a random stream of inputs to hit it. On-the-fly
generation is the efficient solution. You drive some random inputs
to verify the overall behavior of the device. Meanwhile, you're
monitoring the device, looking for a point when you identify the
right conditions, and then you inject the right kind of input to
get the device to the corner case.

Bottom line: You spend less simulation cycles trying to cover
corner cases, and yet you hit them from multiple random scenarios,
resulting in a higher coverage.

2) The need for an exact cycle-accurate model

The proposed approach of having the testbench off-line during
simulation relies heavily on cycle-accurate models. The way it is
described, those models should be capable of modeling everything in
the behavior of the design. This might include complicated aspects
like pipelines, packet-dropping prediction, caches etc. As a
result, whenever any change is made to the design which may have
affected even the most subtle timing-related behavior or ordering
of events, you must now revalidate that your reference model is
still cycle accurate!

In contrast, with the testbench on-line during simulation, the task
is much easier. Objects to represent transaction, scoreboards,
temporal language to describe protocol rules (assertions), and
more, are all at your disposal. Creating a self-checking testbench
is faster, and maintaining it is much easier.

Bottom line: You spend less time developing, debugging and
maintaining the reference model, and instead, focus on verifying
the design at hand.

3) Lack of coverage - knowing what you're missing!

Suppose we adapt the suggested approach, and continuously run
random tests on every available CPU. When do we stop? How do we
know that we have indeed verified everything?

Just using "coverage of the stimulus from the generation engine" is
not enough. Even if you collect coverage on the generated inputs as
Tom West suggests, you still need to be able to cross-correlate
that with other aspects (cross-coverage)-- outputs of the design or
internal white-box elements. For example, knowing that you injected
every kind of packet is critical but insufficient. How about the
routing/control logic? Have we injected each kind of packet while
in each state/mode of that logic? What about the buffers? Have we
injected packets of different lengths while the buffer was empty/
full/almost full?

Bottom line: Without coverage information, you're running "blind"
simulations -- not knowing what scenarios have been tested, and not
knowing whether you're hitting the same few spots over and over
again while missing many others. In fact, I would not be surprised
if it turns out that the 20 CPUs running "blind" simulations may be
getting less actual coverage than what 5 CPUs could if enhanced
with a coverage model. Coverage gives you visibility, and as a
result, makes your verification complete (cover all functionality),
and efficient (avoid simulation cycles that add no coverage).

4) Debugging

What happens when a test case fails? You now need to debug it to
determine the root cause. In the proposed solution with the
testbench off-line, the post-run comparison can only indicate a
clock cycle when the design and the reference model
disagree. However, that may be the end result of a protocol
violation elsewhere, which occurred hundreds or thousands of cycles
earlier, and has propagated to show bad data on some ports. How
much time will you need to spend debugging all such failures if you
only use a waveform?

In contrast, with the testbench on-line you can have e assertions
in place, stopping the test exactly when they detect the violation,
and giving a clear error message of what went wrong. The debug time
saved can be tremendous.

5) Available verification IP

There are many verification components (eVCs) available out there
for every major standard interface. They provide generation,
protocol checking, and functional coverage. The proposed approach
means rendering most such components almost useless, since you can
no longer use their on-the-fly generation, run-time protocol
checking and functional coverage.

An important part of verification efficiency is to be able to have
an environment up and running as fast as you can. Verification IP
allows you to do just that. As a result, verification engineers can
start verifying the core functionality of their design sooner,
rather than focus on the elaborate details of each one of its
interfaces.

All the above aspects are part of a methodology. A methodology that
automates your verification by using a tight interaction between three
key components: Constraint-driven generation, self-checking and
functional coverage. Take one out, and the benefit of the methodology
diminishes. You get less automation and are left with more manual
work. Then you need to consider the cost of engineering time, and more
importantly, the cost of missing your time-to-market window.

We should carefully examine proposals such as these to fully
understand their implications. Even if they increase the amount of
simulations, they may in fact be decreasing the quality of
verification. And after all, is our objective running more
simulations or taping out a bug-free design?

Newsletter · Original Contribution Joined: Dec 08, 2003 Posts: 1107

(Originally from Issue 4.2, Item 5.0)

From: Tom West

> I think this article is missing the point. The point is that
> verification engineers need to verify that their design is
> functioning correctly, and that its complete functionality was
> verified. The question then is, does the proposed solution provide
> that? I think it does NOT.

As much as I respect Zeev, and as much as I am 100% behind every
technical point he made, I don't believe he saw the point to my
article. I think there is a misperception amongst vendors that one
verification language and one methodology fits all applications
equally. In my 18 years of verification, I have learned that there
are many ways to "skin a cat". Specman is a world class verification
tool, no doubt, but is Specman combined with its Verisity recommended
methodology the best verification solution for every application? I
think most of us know this answer. In fact, from "some" of the
response I received via email concerning my article, I helped "some"
find ways to use Specman with an alternate methodology when it would
otherwise NOT have been used at all. The negative reaction I received
in emails from my "friends" was disappointing.

> Vera and VCS simulate with very little overhead as they communicate
> using Direct Kernel Access rather than using the PLI. The speedup
> depends on design style and interface activity, but these run
> typically 50% faster than Vera + NC-Verilog, and speedups of 3x have
> been seen.

From Chris's response, I think he missed the point just a bit on the
issue. The issue is not a PLI vs. single kernel. VCS will not solve
this. PLI overhead is not my concern. The issue is about getting
maximum processing power from your simulation and HVL licenses. The
only way to do this is to NOT run them on the same processor, and NOT
have them running in lock-step with each other. Chris does bring up a
valid argument with his regression license model.

> It's not clear to me why you'd use an HVL if it wasn't for this
> reactive nature. What does an HVL buy you as merely a generator as
> compared to any decent scripting language?

A very good question and one that suggests that an HVL tool may not be
the best for every application. Yet I still contest that the
generation aspect is often reason enough to use the HVL tool.