The Kiuwan Headless Scanner (KHS) has a number of configuration options in the package as we deliver it to the client.
Note: The Kiuwan Headless Scanner may appear in configuration files as Qaking
Main user configurations
The file qaking.user.properties can be found in:
bin\qaking.user.propertiesInclude and exclude patterns
Lines:
include.patterns=
exclude.patterns=
With these lines, you can exclude or specifically include certain source code files in your analysis. In KH,S these lines are empty by default. However, you can use the Kiuwan Local Analyzer properties as an example of common exclusions or inclusions:
exclude.patterns=**/src/test/**,**/__MACOSX/**,**/*.min.js,**/*.Designer.vb,**/*.designer.vb,**/*Reference.vb,**/*Service.vb,**/*Silverlight.vb,**/*.Designer.cs,**/*.designer.cs,**/*Reference.cs,**/*Service.cs,**/*Silverlight.cs,**/.*,**/Pods/BuildHeaders/**/*.h,**/Pods/Headers/**/*.h,**/node_modules/**,**/bower_components/**,**/target/**,**/bin/**,**/obj/**,**/dist/**,**/lib/**These are Apache Ant patterns: **/ means everything recursively before; /** means everything recursively after.
Further Reading: More information on exclusions and inclusions is available here, Source Code Filter Documentation
Timeouts
There are timeouts for maximum analysis engine execution time and for parsing time:
# Timeout to use for maximum analysis execution time, in millisecs (30' = 1800000)
timeout=1800000
# Parsing timeout, in millisecs (zero or negative means no timeout)
parse.timeout=5000
The KHS has been proven to work up to 4 days, so there is no explicit timeout limit. As an example, to set a 4-day or 96-hour limit, you need to calculate:
4 days * 24h/day * 3600 sec/h * 1000 msec/sec = 345600000
Normally, the parse.timeout default value is ok but for extremely large/complex files or a very large code base, this can be increased to, for example, 2 minutes:
parse.timeout=120000
Engine memory allocation
Lines:
# Java virtual machine settings (used only when forked=true)
memory.start=512m
memory.max=512m
stack.size=1536kThe KHSengine process is a Java Virtual Machine (JVM) that needs proper memory allocation. It is difficult to give a fixed relation between the Lines Of Code (LOC) of the source code project you want to scan and the memory you need to allocate. But as a rule, with larger source code projects, you will need to allocate 4, 8, or even 16 GB of memory, and have this memory physically available/free.
For larger projects that take more time to analyze, you can use tools like Java’s “jconsole” to monitor memory use and adjust where necessary. This is especially important when you want to execute several instances of the KHS in parallel on the same machine.
For this, you need to change “memory.max”. If you allocate more memory to the engine for large projects, you can also increase the stack.size to a higher value, such as 4096k.
Source code language-specific configurations
The File qaking.configuration.properties can be found in:
bin\qaking.configuration.properties
The KHS engine has a large number of settings that are specific to each source code language. This is especially important for Cobol. Example (parts):
# ------------- COBOL ----------------
# If set to one of the supported dialects (cobol, cobolibm, cobolmicrofocus, coboltandem, acucobol, rmcobol).
# Leave empty if dialect should be detected
cobol.dialect=
# If true, consider input files as free-format Cobol (no left/right margins)
cobol.transform.freeform=false
# The margin format for Cobol source files.
# Any of autodetect, no_margin, left_margin, right_margin, both_margin, already_transformed, tandem, tandem_ansi, or terminal.
# The default is autodetect, which means that the transformation will try to detect the margins used heuristically.
cobol.transform.marginType=autodetectThere are also settings for when one language is embedded into another, for example:
# ------------- COBOL ----------------
# If true, process SQL embedded in EXEC SQL statements, and include the parsed tree below the EXEC SQL statement node
cobol.processSQL=true
# The parser to use for embedded SQL; could be sql, sqldb2, sqlinformix, or a bean id.
# If not set, use the default for SQL technology (ls_sql_parser_chooser)
cobol.sql.parser=
# ------------- HTML -----------------
# If true, embedded JavaScript code will be parsed, and its parsed tree will be placed below the container script element.
html.processJavascript=true
# ------------- JAVA -----------------
# If true, generate a java class for each JSP page and analyze it with java rules.
# If false, any JSP page is not converted to a java servlet and not analyzed under java technology.
java.processJsp=trueFurther Reading: Read the full qaking.configuration.properties file to see all the available options.
Mapping source code file extensions to languages: file LanguageInfo.properties file
This is the same file and format as in Kiuwan, for the “discovery phase” to map source code file extensions to source code languages.
Can be found in:
bin\LanguageInfo.properties
Example lines:
c=h,ccobol=cob,cbl,cpy,pco# Please note that h and c are shared with c technologycpp=h,hh,hpp,c,c++,cc,cpp,cxx,pccsharp=cshtml=htm,html,xhtmljava=javajavascript=js,jsx,xsjs,ts,tsxYou can edit this file so that any file extension can be analyzed as any source code language that Kiuwan Headless Scanner supports. However, if the files with this file extension are not in the syntax format of the language that it is mapped to, it will probably generate a “parsing error” and will not be analyzed.
Currently, this file contains a number of references to languages that are no longer supported. The supported languages in KHS are those for which there are ruleset XML files in the Scanner’s rulesets folder and on the Kiuwan website.
Changing logging level: log4j.properties file
Can be found in:
conf\log4j.properties
In this file, you can change the level of logging, according to the standard “log4j” definitions:
https://www.tutorialspoint.com/log4j/log4j_logging_levels.htm
The default root level is “WARN”. Lowering this level can produce larger logfiles, but it can be useful for debugging issues.
Changing the default Rulesets
In the Rulesets folder are the ruleset files with the configuration definitions for all rules. These are XML files, which have all the rule definitions for one language in the same file.
For every source code language, there are 2 ruleset XML files:
- qaking_<language>.xml: for Quality scans
- qaking_<language>_security.xml: for Security scans
Both ruleset files have the same XML format. And for most languages, they have a large overlap, or even, the “security” ruleset is just a subset of the rules of the “quality” ruleset. So “security” rules will also be present in the “quality” ruleset, but quality rules will not be present in the security ruleset.
In these XML files, you can change several settings of any rule. The generic format of these ruleset files is, for example:
<?xml version="1.0" encoding="UTF-8"?>
<ruleset name="qaking_CSHARP" updated="">
<description></description>
<rule name="......
</rule>
<rule name="......
</rule>
.....
<categories>
.....You can find the rule you want to change by its rule code, description, or certain keywords. A complete rule definition is inside the <rule ….> … </rule> tags.
Most fields here should not be changed so as not to break the rule’s functionality, but some fields can be changed:
Active / not active
To (de)activate a rule: at the end of the line:
<rule name=...There is the property:
active="true">You can change to:
active="false">Example
<rule name="OPT.CSHARP.Csharp.DefaultParametersShouldNotBeUsed" message="DefaultParametersShouldNotBeUsed: Do not use default parameters" class="com.optimyth.csharp.rules.csharpgen.DefaultParametersShouldNotBeUsed" technology="CSHARP" active="true">Rule category
Rule categories can be changed in two ways:
Change between active=”true”/”false” in “security” and “quality” rulesets
(De)Activate the rule in the “security” and “quality” rulesets. For example, to move a rule from “security” to “quality”, you can deactivate it in the “security” ruleset, and (copy and) activate the rule definition only in the “quality” ruleset. With this, a rule will only be executed in a “quality” scan, but not in a “security” scan.
Category definition inside the rule
Independently, for both the Quality and Security rulesets:
Inside the rule definition, there is a tag that defines the rule’s category, for example:
<idcat>OPT.CRITERIUM_VALUE.CQM.SECURITY</idcat>
This category definition allows categorization of rules by security or quality characteristics, such as “Maintainability”.The rule category appears in the scanner’s output files:
<application name>_<language>.xml.issues
Example
MyApp_javascript.xml.issues
At the beginning of these files there is a section with a summary of rules, violations etc, grouped by category. For example:
<IssuesByCategory>
<Category category='OPT.CRITERIUM_VALUE.CQM.EFFICIENCY' parent='JAVA' technology='JAVA' count='52'>
<Check id='OPT.JAVA.IO.DontUsePrintStackTrace' severity='high' count='10'/>
<Check id='OPT.JAVA.JDBC.PST' severity='medium' count='8'/>
....
</Category>
<Category category='OPT.CRITERIUM_VALUE.CQM.RELIABILITY' parent='JAVA' technology='JAVA' count='308'>
<Check id='OPT.JAVA.CONV.ObjectTypeVerification' severity='medium' count='58'/>
.....
</Category>
<Category category='OPT.CRITERIUM_VALUE.CQM.SECURITY' parent='JAVA' technology='JAVA' count='34'>
<Check id='OPT.JAVA.SEC_JAVA.HttpParameterPollutionRule' severity='high' count='1'/>
<Check id='OPT.JAVA.SEC_JAVA.InformationExposureThroughErrorMessage' severity='high' count='11'/>
.....
</Category>
</IssuesByCategory>These are the possible categories:
OPT.CRITERIUM_VALUE.CQM.EFFICIENCY
OPT.CRITERIUM_VALUE.CQM.MAINTAINABILITY
OPT.CRITERIUM_VALUE.CQM.PORTABILITY
OPT.CRITERIUM_VALUE.CQM.RELIABILITY
OPT.CRITERIUM_VALUE.CQM.SECURITY
Rule priority / severity
Inside the rule definition there is a priority tag., for example:
<priority>3</priority>
The possible values are 1 to 5, and have corresponding text values explained below:
1 = critical
2 = high
3 = medium
4 = low
5 = info / very low
Note: In the KHS output files, this is referred to as severity. In the output files:
<application name>_<language>.xml
Example
MyApp_javascript.xml
In these files, priority/severity is indicated in numbers 1-5. We can see, for example:
<Issues>
<Issue>
<Title>scriptAyudaDpmEstados</Title>
<IssueTypeId>OPT_JAVASCRIPT_ERRORCOMUN_AvoidEmptyFunctions</IssueTypeId>
<Severity>4</Severity> <======== Priority of the rule, as a number
<Files>
<SourceFile>
<File>bin\js\scriptAyudaDpmEstados.js</File>
<Line>180</Line>
.....Also, in the files:
<application name>_<language>.xml.issues Example:
MyApp_javascript.xml.issues
In these files, priority/severity is indicated in words. We can see, for example:
<IssuesBySeverity>
<Severity severity='critical' count='34'/>
<Severity severity='high' count='126'/>
<Severity severity='medium' count='0'/>
<Severity severity='low' count='0'/>
<Severity severity='info' count='0'/>
</IssuesBySeverity>
.....
<Issues>
<Issue id='1' check='OPT.JAVASCRIPT.TYPESCRIPT.UseTypeAnnotations' severity='high'>
.....Repair difficulty (not used in output KHS)
Every rule has defined a level of “repair difficulty” for creating inventories of “work to do”. Inside the rule definition, there is this tag:
<effort>3</effort>
This category definition is an inheritance from the old Checking QA system and is also used in Kiuwan. In the KHS, the possible values are 1..5, referring to:
1 = very hard
2 = hard
3 = normal
4 = easy
5 = very easy
Note: This tag is not used in the output files of the KHS. In Kiuwan, this tag is used to filter violations on for example, those that are considered (very) easy to repair, and to create reports of “technical debt” and “action plans” with estimated working hours needed.
Rule parameter values
There are a number of generic rules that can be customized by changing the rule’s parameter values. With this, the rule can detect more precisely what you want to see.
Example
In this rule for the Java language, the rule parameter value holds a comma-separated list of library names that should not be imported. In ruleset file qaking_java.xml:
<rule name="OPT.JAVA.RGP.AvoidPackages" message="AvoidPackages: Do not import the specified packages" class="com.als.clases.port.rgp.AvoidPackages" technology="JAVA" active="true">
.....
<properties>
<property name="packages" descname="Comma-separated list of regular expressions for packages to detect." value=""/>
</properties>Although the default value is empty, the client can add unwanted deprecated or proprietary libraries, for example:
<property name="packages" descname="Comma-separated list of regular expressions for packages to detect." value="java.security.Identity,java.mycompany.mylibrary"/>With this changed parameter value, this rule will emit a violation for every “import” of the libraries in the parameter value.
Do not change these rule definition tags
These tags in a rule definition are fixed and should never be changed:
<rule name="<rule ID>" ..... class="<internal Java class with rule's functionality>" technology="<language>" .....>
Example
<rule name="OPT.CSHARP.AvoidNullReferenceException" ..... class="com.optimyth.csharp.rules.csharpoyr.AvoidTypeNamesInParameters" technology="CSHARP" .....><property name="<fixed rule parameter name>" descname="<fixed rule parameter description>" ..... />
Example
<property name="typesToCheck" descname="Language-specific type names." ..... />Custom configuration files
The Kiuwan Headless Scanner has the ability to override a number of configuration and scanning options by using custom-created configuration files. These can be global or on a per-scan basis.
The original documentation about this is in the following file in the KHS installation:
doc\README.customconfig.txt