System State Backups broken on Windows 2012 R2 ?

Avast Business Cloud Management Console & Clients
1

Hi All

I’m trying to diagnose a problem with System State backups on a Windows 2012 R2 server that has Avast for Business Cloud (Free).

Some time ago, the Windows July 2016 update rollup KB3172614 was installed but somehow was corrupted and caused a number of issues with the server, one of which was an issue successfully starting a System State backup.

Since then, the patch has been fixed but the System State backups fail for a different reason (different error) than previously. I have diagnosed the probable cause to be an bad enumeration of service imagepaths in the registry, and all the problematic paths seem to be related only to Avast service files.

The Avast program version was upgraded before the Windows patch issue was fixed, so I am wondering if the program upgrade introduced a new problem that I didn’t have using the older Avast program version (ie when System State backups was still working). Since I had the problem with the patch for so long, it’s difficult to say if this alternate error was introduced by the Avast update.

So, rather than bombard you with error logs and technical detail, my question is simply does anyone else use the latest AfB cloud (free) on Windows Server 2012 R2 and can still perform successful system state backups? I don’t want to go reinstalling just now (production server), particularly if it turns out to be a bug.

Look forward to any results. 8)

2

I jumped ahead a little and found a VirtualBox image of Windows Server 2012 R2 and tested my theory. Here’s my results:

Vanilla Windows Server install: System State backup works
Installed Avast Business Cloud 17.2.3419: System State backup fails to enumerate files
Uninstall Avast: System State backup works
Installed Avast Business Cloud 12.3.2515: System state backup works

I hope there are others out there that can validate this so I can report it as a bug with some evidence to back up my testing :slight_smile:

3

Hi, one of our sub-resellers found this issue about 1 month ago in some of his customers. We communicated the problem then, using the support system, but due to the tests that they asked us to perform and to certain problems that were with the ticket system, the matter has been delayed more than anticipated.

Finally, yesterday we have received a confirmation that the issue has integrated into the system with an issue-ID (AV-13802). I hope this will mean that will be solved soon.

In any case, maybe it would be interesting to write to support providing your own data and evidence, and mentioning this thread and the ID I said.

4

Thank you Juanjo, glad to know its not just me going crazy ;D

Will definitely contact support and see what they say and use this issue ID as reference.

Did you happen to get a workaround for this?

If not, I’ll see if I can figure one out. I’d really like to get this server back on track after the mess the bad Microsoft patch left behind. Like you say we might be waiting some time for a binary fix. I’m surprised there aren’t more reports of this problem.

5

Hi:

No, we don’t have a workaround right now, sorry.

“I’m surprised there aren’t more reports of this problem” → same here!!!

6

I contacted support and they confirmed the issue too, so I asked if there was a workaround and this was the reply…

Unfortunately, there is not a work around at the moment. However, with the soon release of the new version the bug will be fixed.

No workaround? Hogwash :slight_smile:

I was also told no ETA on new version… oh please you can do better, Avast. Make a guess. 1 week, 1 month, 1 year?

So the problem I experience is the system state backup errors while enumerating a list of files to backup and gets part-way into a backup of a poorly enumerated path and fails with Error in backup of C:\windows\systemroot\ during enumerate: Error [0x8007007b] The filename, directory name, or volume label syntax is incorrect.

The workaround I suggest for anyone experiencing this problem is to uninstall v17.2 and install an earlier version. I tested with v12.3 and system state backup still works on my virtual test host. Make sure if you do this that the host is using a cloud settings template that does not perform automatic program updates or it will just upgrade again :slight_smile: If system states still do not work for you, then there might actually be another problem with the host which is not possible to diagnose in this forum. Remember, always test on non-production first for your specific environment!

If you don’t have an earlier version and cannot obtain one from support (usually they can supply if you need it) and cannot wait for a new version, the other workaround I found is modifying the HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services registry. Because of the dangers here, I have opted not to provide specific instruction on how to change this. Doing an incorrect backup and edit of this part of the registry can make your system unbootable and is for experienced systems admins only. You have been warned, and I take no responsibility for any losses you might suffer by using the information I provide here! You’ll also need to reboot for this registry edit to take effect and so downgrading the program version is just as disruptive on a production host and is why it is probably the better (supported) choice.

Here are the following keys that I discovered cause the system state to fail:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswbidsh
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswblog
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswbuniv
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswRvrt
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswVmm

You either need to change the ImagePath values to have a fixed path before the filename (eg C:\windows\system32\drivers\aswbidsha.sys), or, a more correct fix, to change the data type of the current ImagePath value from REG_SZ by deleting the value and recreating it as a REG_EXPAND_SZ.

I’m no expert and can’t explain why loading on boot (ie, the value Start=0) does not have a problem with the current REG_SZ but system state does. And so for that reason a program downgrade is still my preferred choice as I expect when the fix is released in an upgrade it is going to upgrade properly.

I hope someone finds this useful.

7

Only a few words to say thank you very much for sharing all this useful information. Very interesting investigation and workarounds.

8

I have the same problem. I have Windows Server 2012 R2 and Avast for Bussines v17.4.2520.

I had discovered, watching the logs, that the faulty Avast driver was aswvmm.sys.

Excuse my english, i don’t write it very well.

Look forward to an answer from avast support.

Greetings from Argentina.

9

Yep, I agree, it still fails, although I think on brief inspection it is only affected by aswRvrt and aswVmm now and not all the ones previously listed. So I’m not sure if the developers actually did try to address their bug, or if their testing is just inadequate.

I’ve emailed Tech Support back to ask what’s going on, I suggest anyone experiencing this problem to do the same as the resolution to this really should be a high priority. Breaking a built-in OS component is bad enough but to then fail to fix it as promised is almost inexcusable.

Welcome to the forum @Maxi

10

Response from Tech Support:
This is a known issue that the development team is working on resolving at this time. We should have a fix pushed out with the next version update within the next business week.

Nice quick response too, I haven’t lost all faith :wink: If it’s true, it sounds like it is being taken seriously. Considering they are planning another update soon there might be a couple of other serious bugs in the current version they are looking to fix too. Might hold off on my full site deployment for now :slight_smile:

11

Well, at least 1 business week has passed, and no new version … not surprised really :stuck_out_tongue:

I contacted support again and was told the bug is still being worked on. So I guess I am just being told what they think I want to hear.

They did ask me to gather debug logs and send them in to be attached to the bug because they said: “the more we report a bug the higher it goes on the priority list”.

So if you are experiencing this issue and have not done so already, contact support to create a case number, and ask to be added to the bug and request instructions on how to upload debug logs. The more people report it, the sooner it will get fixed… maybe :smiley:

12

For those interested, the 17.5 release hasn’t fixed the issue. I asked support about the current status and they said:

The developers have identified the specific error that causes this problem, and the fix was sent to QA for review yesterday. We don’t have a release date for this fix just yet, but it should pass QA review and be included in our next major release (17.5) or the next release after that (17.6).

Since 17.5 was released before the QA review, it should be coming in the following release. I haven’t seen a consumer beta yet, so that might still be a few months away. :-\

Unfortunately the only workarounds I know are either:

  • keep running Avast 12.3 and ignore the danger message in the console (my current preference)
  • do a full volume backup of the system drive without System State selected (but can complicate restore types)
  • use a 3rd party tool
  • use my registry fixes above (not recommended)

I’d be keen to hear about how others avoid the problem, eg what 3rd party tools they use for System State backups. :slight_smile:

13

I have got the same issue. I have got 17.6 on a server and it is still an issue.

I have just contacted support about it as well.

Steve

14

Yes I can confirm this still occurs on my test server too.

Thanks for logging it with support. The more people that report it, the higher priority it gets in being fixed.

15

Its a real pain. I have got Avast on about 15-20 servers currently.

Steve

16

Are all your servers 2012 R2 or this affects other server versions you have too? I assumed it was just a 2012 R2 issue given the low priority Avast have put on the bug.

17

I have got the issue with Server 2016, Server 2012r2, Server2012. I had contact from Avast last Tuesday/Wednesday but nothing since.

Steve

18

i have the same problem on 2008 r2, 2012r2 and 2016 servers. Is there any stable workaround?

19

No, there is no official workaround, however downgrading to v12 works for me. Please log this fault with Avast support so that they prioritize this bug fix. The more reports they get the faster they will fix it, or so they tell me :-X

20

This works (with disable selfprotection) for now at least

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswRvrt]
"ImagePath"="system32\\drivers\\aswRvrt.sys"

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aswVmm]
"ImagePath"="system32\\drivers\\aswVmm.sys"