Microsoft Dynamics CRM Team Blog: CRM Usage Reporting Unleashed

[Blog bot]

Источник: http://blogs.msdn.com/crm/archive/20...unleashed.aspx
==============

Our CRM MVP guest blogger for today is David Jennaway who is the technical director at Excitation.

A frequent request we come across is from companies who want to know which users are using CRM and when. The CRM platform provides the facility to gather detailed usage information by writing plug-ins, but a simpler and more general mechanism is to use the Internet Information Services (IIS) logging mechanism.

This article will explain how to derive useful usage metrics from the information provided by IIS logging. The main steps are:

1. Configure IIS logging to allow easy querying of the log data

2. Filter the raw log data to help identify usage patterns

3. Specify time periods to help categorize when people access CRM

4. Present the usage information in a graphical format through reports

Configure IIS logging

IIS logging allow the capture of information about every web request submitted to the server, and it is configured at the web site level. By default the log information is written to text files, but IIS can also write the information directly to a relational database, which allows easier analysis. For the purposes of this post, the simplest setup is to configure IIS 6.0 to write the log data to a SQL Server ODBC data source (links to other options are given below).

To configure an IIS 6.0 server to write log information for the CRM web site:

1. Create a SQL Database. In this database, create a SQL table to hold the log data. IIS has a SQL script that will create the table with appropriate columns – this script will be in %windir%\system32\inetsrv\logtemp.sql

2. Create an ODBC data source on the IIS server with the connection details to the database in step 1. The data source must be a system DSN, and I’d recommend using Integrated authentication to SQL Server

3. In IIS Manager, open the Properties of the Microsoft Dynamics CRM web site. On the Web Site tab, select ODBC Logging in the Active log format dropdown, and set the associated properties to reflect the SQL table and ODBC DSN created in steps 1 and 2. If you specified Integrated authentication in the DSN, then the User name and Password will be ignored, though oddly, you cannot leave the password field blank

Related links:

• For more information on the above, and more detailed setup instructions, see http://support.microsoft.com/kb/245243
• Setting up ODBC logging on IIS 7.0 is a little more involved. See http://blogs.iis.net/rakkimk/archive...c-logging.aspx

Filtering the log data

The IIS log will have a record for every request submitted to the web site. This includes requests for images, stylesheets and other supporting files, and can result in more data than you need . For instance, opening a CRM record can result in 30 records in the log table. There are also entries for CRM processes (such as the CRM Asynchronous Service) accessing the CRM platform.

This extraneous data can be filtered out with a SQL view. We use the following view definition:

CREATE view [dbo].[IISLogFiltered]
as
select ClientHost, username, LogTime, processingtime, bytesrecvd, bytessent, servicestatus, operation, target, parameters
from inetlog
where username not in ('-', 'CRMTEST\CRM4DEV$')
and parameters '-' and parameters not like 'lcid=%'

-- Replace CRMTEST\CRM4DEV$ with the account used by the CRM services

This assumes the SQL table has the default name (‘inetlog’), and filters out anonymous requests, and those from the CRM services. It also filters out requests with no query string parameters (or where the only parameter is ‘lcid’), which excludes the requests for the supporting files. You can explore the log data in more detail, and adjust the filters as appropriate.

Specifying time periods to categorize access

Even with the SQL view described above, you are still a step away from getting useful information about when users access CRM. The raw data stores the date and time of each request in one field, but it will help to process this into time and date periods. The approach we use is to build a SQL table that contains definable time periods , and a SQL function to match the log time to a time period. The SQL table can be created with the following script:

CREATE TABLE [dbo].[TimePeriod](
[PeriodText] [nvarchar](20) NULL,
[HourStart] [int] NULL,
[MinuteStart] [int] NULL,
[TotalMinuteStart] AS ((60)*[HourStart]+[MinuteStart]),
[TotalMinuteEnd] [int] NULL)

You can then populate it with data for the time periods you want, for example:

PeriodText

HourStart

MinuteStart

TotalMinuteStart

TotalMinuteEnd

00:00 - 08:00

0

0

0

480

08:00 - 09:00

8

0

480

540

09:00 - 10:00

9

0

540

600

10:00 - 11:00

10

0

600

660

11:00 - 12:00

11

0

660

720

12:00 - 13:00

12

0

720

780

13:00 - 14:00

13

0

780

840

14:00 - 15:00

14

0

840

900

15:00 - 16:00

15

0

900

960

16:00 - 17:00

16

0

960

1020

17:00 - 18:00

17

0

1020

1080

18:00 - 00:00

18

0

1080

1440

This splits the day into hour-long periods between 8:00 and 18:00, with one period before and after the hour-long periods. The table can be used to find the PeriodText from a datetime field using a SQL function:

CREATE function GetTimePeriodText (@DateTime datetime) returns nvarchar(20)
as
begin
declare @tm int, @ret nvarchar(20)
set @tm = 60 * datepart(hh, @DateTime) + datepart(mi, @DateTime)
select @ret = max(PeriodText) from TimePeriod where @tm >= TotalMinuteStart and @tm < TotalMinuteEnd
return @ret
end

It will also help to extract the date component, this can be done using the SQL Convert function. The following SQL view applies these functions to the above SQL view used for filtering. It also extracts the CRM organization name from the target (url), and removes the domain part of the username.

Create view IISLogFilteredWithPeriods
as
select ClientHost, username, LogTime, processingtime, bytesrecvd, bytessent, servicestatus, operation, target, parameters
, dbo.GetTimePeriodText(LogTime) as TimePeriod
, convert(nchar(10), LogTime, 103) as DateText -- The last parameter defines the format
, convert(nchar(8), LogTime, 112) as DateYYYYMMDD -- Useful for sorting dates
, case when charindex('/', target, 2) > 2 then substring(target, 2, charindex('/', target, 2) - 2) else '' end as Organization -- Get organization name from target
, rtrim(substring(username, 1 + charindex('\', username), len(username) - 1 + charindex('\', username))) as usernameonly -- Remove Domain part of user name
from IISLogFiltered

Note that the IIS log will store data in Universal Time (UTC). You can modify either the data in the TmePeriod table, or the logic in the GetTimePeriodText function, to apply timezone information. Another, though officially unsupported approach, would be to use the fn_UTCToLocalTime SQL function in the MSCRM database.

Creating reports on the log data

Now we’ve got the structure to process the log data, we can present in to users via a SQL Server Reporting Services report. The layout is up to you, but we find an effective way to present the information is in a matrix, with the time periods along the columns, with conditional formatting to highlight periods of light or heavy use. The following is an example, using the report definition below:

     

The numbers are the count of requests, with the background colours indicating how heavy the use is (as it’s a test system I’m considering 5 or more requests per hour to be heavy usage).

The SQL statement that produced this report is:

select usernameonly as username
, TimePeriod, DateText, DateYYYYMMDD
, sum(processingtime) as ProcessingTime, count(*) as Requests
from IISLogFilteredWithPeriods
where logtime > dateadd(d, -7, getdate())
group by usernameonly, TimePeriod, DateText, DateYYYYMMDD

For performance reasons, and out of habit, the aggregation is done in the SQL query, rather than in reporting services.

The full report definition follows at the end of this post.

Further thoughts – performance

This post focused on providing an overall process for getting usage information from the IIS logs. There are some performance overheads when using ODBC logging - it is more processor intensive that logging to file or raw formats, and it affects the IIS caching mechanism.

An alternative approach is to log to raw or file formats, and either import the data periodically to SQL Server (see http://support.microsoft.com/kb/296093/ and http://support.microsoft.com/kb/296085/ ), or process it with other tools, such as the LogParser tool (http://www.microsoft.com/downloads/d...displaylang=en)

You should also consider archiving or deleting old log data. One heavy user of CRM can easily generate over a megabyte of log data per day, so the quantity of data can mount up quickly.

If logging directly to SQL Server, there is a trade-off to consider with respect to SQL indexes. Adding a non-clustered index on the parameters column may help with the analysis performance, but would adversely affect the logging performance. A clustered index on the logtime column is recommended for all circumstances.

Further thoughts – more analysis

The report provided in this post is limited to displaying the number of request per user. Further ideas for analysis are:

• Using the processingtime data to get an idea of the processing on the IIS server. This is a measure of the elapsed time from starting processing the request in IIS, to submitting the end of the response, and will be affected by the load on the IIS server. This can be useful for identifying when and if requests take a long time to complete
• Identifying the types of entities accessed. The target can help identify which entity has been requested. This takes a bit of parsing, but the topic ‘URL Addressable Forms and Views’ (http://msdn.microsoft.com/en-gb/library/cc150850.aspx) in the CRM SDK will help
• IIS logs are not useful for identifying the type of data operation (e.g. create, update, delete, assign) has been performed. If you need this information, you would be advised to use a plugin/callout mechanism for the auditing – see http://blogs.msdn.com/crm/archive/20...usage-log.aspx for an example of this
• The techniques described in this post were developed for CRM 4, but also work on CRM 3. The only difference is that there is no need to extract the Organization name from the target on CRM 3

Source files

The SQL objects, and report RDL described in this post are available for download on the MSDN Code Gallery here (http://code.msdn.microsoft.com/IISLoggingCRM). I’d welcome any comments there, and I can add people as contributors to the code gallery resource if they have further reports or SQL objects they are willing to share.

Full report definition

This is a SQL 2005 Reporting Services report. The Data Source will need to be changed for your environment:

1: 2: 3: 4: 5: c04397fa-f432-4983-907e-3533dc2dea9c 6: 7: SQL 8: Data Source=crm4dev;Initial Catalog=IISLog 9: true 10: 11: 12: 13: 11in 14: true 15: 8.5in 16: 0.25cm 17: true 18: 2.5cm 19: 2.5cm 20: 2.5cm 21: c13a6342-b76c-4be0-8d0f-8598eac7fa80 22: 21cm 23: 24: 25: 26: 27: username 28: System.String 29: 30: 31: TimePeriod 32: System.String 33: 34: 35: DateText 36: System.String 37: 38: 39: DateYYYYMMDD 40: System.String 41: 42: 43: ProcessingTime 44: System.Int32 45: 46: 47: Requests 48: System.Int32 49: 50: 51: 52: IISLog 53: select usernameonly as username, TimePeriod, DateText, DateYYYYMMDD 54: , sum(processingtime) as ProcessingTime, count(*) as Requests 55: from IISLogFilteredWithPeriods 56: where logtime > dateadd(d, -7, getdate()) 57: group by usernameonly, TimePeriod, DateText, DateYYYYMMDD 58: true 59: 60: 61: 62: 16.5cm 63: 64: 1cm 65: 66: 67: textbox1 68: 69: SteelBlue 70: Tahoma 71: 20pt 72: 700 73: 2pt 74: 2pt 75: 2pt 76: 2pt 77: 78: 1 79: true 80: 0.91429cm 81: Log By Date And User - Last 7 days 82: 83: 84: 85: 86: 1.35cm 87: 88: 89: IISLog 90: 91: 92: 2.53968cm 93: 94: 95: 96: =Fields!DateText.Value 97: 98: 99: 100: 101: =Fields!DateYYYYMMDD.Value 102: Descending 103: 104: 105: 106: 107: 108: textbox5 109: 110: White 111: #6e9eca 112: 113: LightGrey 114: 115: 116: Solid 117: 118: Tahoma 119: 700 120: 2pt 121: 2pt 122: 2pt 123: 2pt 124: 125: true 126: Total 127: 128: 129: 130: 131: 132: DateText 133: 134: White 135: #6e9eca 136: 137: LightGrey 138: 139: 140: Solid 141: 142: Tahoma 143: 700 144: 2pt 145: 2pt 146: 2pt 147: 2pt 148: 149: 3 150: true 151: =Fields!DateText.Value 152: 153: 154: 155: 156: 157: 4.5cm 158: 159: 160: 161: =Fields!username.Value 162: 163: 164: 165: 166: =Fields!username.Value 167: Ascending 168: 169: 170: 171: 172: username 173: 174: White 175: SlateGray 176: 177: LightGrey 178: 179: 180: Solid 181: 182: Tahoma 183: 700 184: 2pt 185: 2pt 186: 2pt 187: 2pt 188: 189: 2 190: true 191: =Fields!username.Value 192: 193: 194: 195: 196: 197: 0.91429cm 198: 199: 200: 0.53333cm 201: 202: 203: 204: =Fields!TimePeriod.Value 205: 206: 207: 208: 209: =Fields!TimePeriod.Value 210: Ascending 211: 212: 213: 214: 215: 216: textbox4 217: 218: White 219: #6e9eca 220: 221: LightGrey 222: 223: 224: Solid 225: 226: Tahoma 227: 700 228: 2pt 229: 2pt 230: 2pt 231: 2pt 232: 233: 4 234: true 235: Total 236: 237: 238: 239: 240: 241: TimePeriod 242: 243: White 244: #6e9eca 245: 246: LightGrey 247: 248: 249: Solid 250: 251: Tahoma 252: 700 253: 2pt 254: 2pt 255: 2pt 256: 2pt 257: 258: 5 259: true 260: =Fields!TimePeriod.Value 261: 262: 263: 264: 265: 266: 9.73968cm 267: 268: 269: 270: textbox3 271: 272: 273: LightGrey 274: 275: 276: Solid 277: 278: Tahoma 279: 2pt 280: 2pt 281: 2pt 282: 2pt 283: 284: 6 285: true 286: 287: 288: 289: 290: 291: 292: 293: 0.53333cm 294: 295: 296: 297: 298: textbox2 299: 300: =iif(Fields!Requests.Value > 5, "LightSteelBlue", iif(Fields!Requests.Value > 0, "AliceBlue", "White")) 301: 302: LightGrey 303: 304: 305: Solid 306: 307: Tahoma 308: 2pt 309: 2pt 310: 2pt 311: 2pt 312: 313: 1 314: true 315: =Sum(Fields!Requests.Value) 316: 317: 318: 319: 320: 321: 322: 323: 324: 2.51428cm 325: 326: en-US 327: 2.5cm 328: 29.7cm 329:



Cheers,

David Jennaway



Источник: http://blogs.msdn.com/crm/archive/20...unleashed.aspx