Use jArchi to draw Archi relations from Azure service map

This blogpost is about a script…

Azure Service map

Azure service map is a really cool thing.  It basically shows you all network connections your servers make when running.  You can view this information in a table or in a visual form.

Picture taken from: https://docs.microsoft.com/en-us/azure/azure-monitor/insights/service-map
Out of the box service map functionality.  The script we discuss here will create something similar directly in Archi..

Now, does this look interesting?  Sure it does, I see the sever, the connections it makes to other servers, I see the processes that start the connections…  all of that in an easy to use interface.  When migrating to cloud this is really valuable information.  When doing firewall projects or management also.  It helps you decide if you should open that port or not.  There are more cases where this information is a huge time saver.

I am not going to write on how to setup this, the microsoft documentation is clear and up to date.  Read all about it here:  https://docs.microsoft.com/en-us/azure/azure-monitor/insights/service-map.  You need 2 agents installed.  Alternatively, if you already run SCOM, you don’t need to install the agents, you can link up SCOM to Log Analytics

TLDR: you install the agents agent on your Windows or Linux sever, you create a log analytics workspace in Azure, you connect both… et voila.

 

Archi

While I am a huge fan of Log Analytics, this blogpost is about using jArchi to draw relations.  As a regular user of Archi you probably have created drawings where you linked servers & applications to each other to form something that supports a business proces.  Service maps would definitely be a valuable source of information for that, because it shows you what application processes (application components) on what server (node) are in place.

Last year the guys that build Archi released a Javascript based engine so that you can script your model.  While they don’t include it in the basic (free) package you can get it when you have made a donation.  Find all about jArchi and getting it here: https://www.archimatetool.com/blog/2018/07/02/jarchi/

Once you have the plugin, you can work with it using the information provided here: https://github.com/archimatetool/archi-scripting-plugin/wiki

This blogpost is about a script, a script for jArchi..

The script

The script will use log analytics data to draw an Archi model with nodes and relations.  it will show you information on the relation as well.

You can find the script here: https://gist.github.com/gevaertw/5048676da755dccfabe1947050b24fc8

I believe it is well documented, but I can understand it is complex.  It is probably not the first script you will try to understand when you just installed jArchi.  Most of it is plain javascript the rest is specific jArchi stuff.

How dos this work?

So, how can you use it?

Log analytics

The starting point is that you have a running setup of Azure service map, sending data to a log analytics workspace.

We need to get the data out of log analytics so that we can import it into the model.

LAPortal2
This is the log analytics dashboard in Azure, after selecting the workspace.
  1. Open the query editor
  2. Enter the query
  3. Run the query
  4. Export the data, make sure to select only displayed columns

 

The log analytics query is included in the jArchi script, meaning; if I ever publish a new  version of the script, an matching log analytics script is included.  You find the log analytics script in the script code like this:

------Start of log analytics query-----------------------------------

VMConnection

| where Direction == "outbound"

| where DestinationIp != "127.0.0.1"

| distinct Computer, ProcessName, SourceIp, DestinationIp, Protocol, DestinationPort, RemoteDnsCanonicalNames, RemoteCountry)

-------End of log analytics query------------------------------------
You can safely change the where clauses, but the result columns (distinct….) should remain the same.
At this point you have a CSV file that contains a lot of records.  Each record shows a connection logged.
You might notice that:
  1. Lots of RemoteDnsCanonicalNames are not filled in.  This is because service map could not resolve them.  Still.. not a problem, we have the remote IP, the script will create a node based on the remote IP instead of the name
  2. Lots of RemoteDnsCanonicalNames point to different hosts in the same network.  In my case there were 50+ different CDN hosts, followed by a similar number of AWS hosts.  None of the names made any sense at all.. 

To archi

Now, as said before this script uses jArchi.  Make sure you have it installed in Archi..

Archi1
Run the script from the script manager

 

Archi2
Select the CSV

 

Now, depending on the number of lines you have in your CSV this might take a while.  On my average laptop a CSV with 1900 lines takes 90 seconds or so.

 

Arhi10
Processing takes some time.
Archi7
When everything is ready you have all nodes and relations as detected by Log Analytics

The script adds some properties, it indicates that the relation or node was created by the script, it also adds a timestamp.  In case of the relation it also adds the origination process.  In a later version I might use an application component, but for now let’s keep it simple.

Archi8
Dropping the nodes on the canvas immediately shows the relations between the objects.

Work efficiently

Our 1900 lines of log analytics data generated an Archi model of around 1 MB.  Models this “big” might no perform well anymore.  You can reduce the number of lines by:

Filtering connections.

Some traffic is known and default.  Your monitoring servers, SCCM, AD…  You know them, so filter that traffic out using the where clauses.  You can also filter out protocols like DNS, the same here, you know it is there, it brings no value

Grouping destinations.

With heavily connected systems you will find al lot of unknown IP addresses on the internet.  Depending on your needs the specific address is not interesting.  group them all into a node called “internet” or so.  The same for connections to pubic cloud providers and CDNs

Removing the process, or port

Depending on your needs the source process is interesting or not.  If you don’t need it, remove it from the file.  The same for processes and ports.

Excel

Excel is great for that, and more.  When you cleaned your file, use excel to remove all duplicate records. Keep in mind that Excel might change the delimiter based on your settings.  The script expects a comma as delimiter.

Wrap up & Conclusion

Hopefully you can use this script to help you build your inventory.  You should however not see this as a replacement for service map itself, it lacks the performance and feature wise I cannot beat Microsoft (and I have no ambition nor desire to do that).  It is meant als the bridge between Log Analytics and Archi, that’s no more, no less.  Feel free to drop me a comment.  I might add some extra features later on, based on what I need in my projects, or your cool suggestions.  I cannot give support however…

Advertisements

2 thoughts on “Use jArchi to draw Archi relations from Azure service map

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s