chevron-thin-right chevron-thin-left brand cancel-circle search youtube-icon google-plus-icon linkedin-icon facebook-icon twitter-icon toolbox download check linkedin phone twitter-old google-plus facebook profile-male chat calendar profile-male
0 votes
Hi

As the title suggests, we are experiencing significantly reduced performance of xml serialization when TM is enabled. This is true even though when Isolator is not used in any of the tests, yes in fact even when no TM assemblies are referenced from the projects involved.

Simple example to prove the point:

using System;
using System.Diagnostics;
using System.Globalization;
using System.IO;
using System.Xml.Serialization;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace TypeMockPerformanceProblem
{
    [TestClass]
    public class UnitTest1
    {
        private static readonly XmlSerializer Serializer = new XmlSerializer(typeof(A));

        [TestMethod]
        public void TestMethod1()
        {
            var stopwatch = new Stopwatch();
            stopwatch.Start();
            var stream = new MemoryStream();
            Serializer.Serialize(stream, new A { Prop1 = "Shalom aleichem", Prop2 = "As-Salāmu `Alaykum" });
            stream.Seek(0, SeekOrigin.Begin);
            var a = (A)Serializer.Deserialize(stream);
            stopwatch.Stop();
            Console.WriteLine(string.Format(CultureInfo.InvariantCulture, "Creating serializer, serializing and deserializing: {0}", stopwatch.Elapsed));
        }
    }

    public class A
    {
        public string Prop1 { get; set; }
        public string Prop2 { get; set; }
    }
}


Run this with and without TM enabled and you will notice that the performance hit is approx 100%. This is very simplified example and the effect can of course be much bigger depending on the usage of xml serialization. In some extreme cases we have seen a couple of seconds turn into minutes(!).

The total impact of this phenomenon on the performance of our ~15K test harness is not known but we have good reason to think it is enourmous (to be conservative ;-)).

Question: Is this something you guys at TM are aware of and if so do you have plans to do something about it? Is it possible without rebuilding the whole thing (I am guessing this is an unwanted consequence of hooking into the profiling interface)? I am really keen to know the answer to these questions since we are now embarking on a quest to seriously speed up our test runs. One of the options on the table is of course to get rid of TM all together, and you wouldn't want that now would you? ;-)

Regards
Martin
asked by brumlemann (2k points)

7 Answers

0 votes
Hi Martin,

Thanks for the report, I checked it out and indeed there is serious performance hit in the code that you posted.
We'll investigate and let you know the results.
answered by ohad (35.5k points)
0 votes
Hi Martin,
If you don't fake in your tests types from System.Xml you can order the Isolator to ignore all the types in that namespace.
To do that open the file blacklist.dat (should be empty by default) in Isolator installation folder and add the line:
System.Xml

This should cut ~50% of the time in example you posted.
Please let me know if it helps.
answered by ohad (35.5k points)
0 votes
Hi Ohad

Good. We will verify and analyse and more tomorrow and in the days to come.

On a general note. It would be way more sweeter to be able to configure stuff like this at runtime instead of setting them in a global context such as blacklist.dat. The same goes for whether Isolator should be enabled at all. It's impossible from the build system to know in advance. Some of our test assemblies doesn't use TM at all but since MSBuild cannot know which we have to enable it for all. The alternative of keeping lists (MSBuild ItemGroup) is clearly not manageable. Think about it ;-)

[AssemblyInitialize]
public static void AssemblyInitialize()
{
Isolator.Initialize();
Isolator.BlackList(new []{"System.Xml",...})
}

This would then be sufficient for Isolator to work. No need for <TypeMockStart/> or <TypeMockStop/>.

(don't know if the prototype above is correct, just about to go home for the day and just scraped it from memory but the point should be clear)

Regards
Martin
answered by brumlemann (2k points)
0 votes
Hi Martin,
Yes it would be nice to do that via code :) but consider this:
1. Every method is jitted only once.
2. During jit time Isoaltor is doing its work.
3. If the code that you want to ignore is called before the code that tells the profiler to ignore the code it will not work.
answered by ohad (35.5k points)
0 votes
Hi

So, we have implemented the suggested action. Some feedback on the gains achieved is in order.

After including System.xml and System.Data in blacklist.dat (and fixing a few tests that actually faked types from these namespaces :-|) the gain on our complete test harness (some 13K tests at the last count) is in the 15% range (from 6.2 hours to 5.2). Nice :-)

Including other system namespaces created more havoc and didn't present the same gains so we have stuck with the two mentioned above.

Thx Ohad. TM leaves to fight another day ;-)

Martin

PS We have implemented some other improvements (to our own test frameworks and their usage) and gained another two hours. Nice*2 :-). And still we haven't been able to take full advantage of our MSBuild parallel execution trick to it's full extent but that's in the making. The last run I did was actually under two hours. Needs verification though.
answered by brumlemann (2k points)
0 votes
Hi Martin,

Thanks for coming back with those stats. Can you share what you did to take the additional 2 hours off?

Thanks,
Gil
answered by gilz (14.5k points)
0 votes
Hi Gil

The measures applied to shave the extra time off are very specific to our own test frameworks but basically we went through all of our integration tests and made sure each and every one runs in a TransactionScope() without scope.Complete() so in essence the changes done by test is rolled back leaving any test databases in pristine state. It can then be reused by the other tests without fear of bleeding.

Next step was to enable reuse of the test databases (a feature we already had in our test frameworks but not all developers were familiar enough with to use to the full extent). Setting up test databases is an expensive task and is now done at a most lower rate.

A lot of grunt work but fortunately Arnulf took one for the team so I am happy. He sometimes refer to the Architect role as being more like a janitor. I have no problems agreeing with him ;-)

Martin
Software Architect
answered by brumlemann (2k points)
...