All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Properties Friends Macros Groups Pages
Public Member Functions | Public Attributes | List of all members
BaseTest.BaseTest Class Reference

Inherited by newFormat.abort_event_test.Test, newFormat.aida2root_test.Test, newFormat.aida2rootex_test.Test, newFormat.algerraud_test.Test, newFormat.algorithn_destructor_test.Test, newFormat.algsequencer_pyopts_test.Test, newFormat.algsequencer_test.Test, newFormat.algtools2_test.Test, newFormat.algtools_pyopts_test.Test, newFormat.algtools_test.Test, newFormat.algtypealiases_test.Test, newFormat.array_properties_test.Test, newFormat.auditors_user_timing_test.Test, newFormat.boost_array_properties_test.Test, newFormat.bug_30209_namespace_in_sequencer_test.Test, newFormat.bug_34121_tool_properties_in_python_test.Test, newFormat.bug_35913_test.Test, newFormat.bug_38882_test.Test, newFormat.bug_41136_test.Test, newFormat.cmdline_conf_test.Test, newFormat.colormsg_test.Test, newFormat.configurableuser_test.Test, newFormat.counterex_py_test.Test, newFormat.counterex_test.Test, newFormat.countersvcalg_test.Test, newFormat.errorlog_test.Test, newFormat.event_timeout_abort_test.Test, newFormat.event_timeout_test.Test, newFormat.evtcolread_test.Test, newFormat.evtcolwrite_test.Test, newFormat.extended_properties2_test.Test, newFormat.gaudicommontests_test.Test, newFormat.gaudimt_test.Test, newFormat.gaudipython_mix_test.Test, newFormat.gsltools_test.Test, newFormat.histo_ascii_dump_test.Test, newFormat.histoex1_test.Test, newFormat.histoex2_test.Test, newFormat.histoex_test.Test, newFormat.histograms_test.Test, newFormat.history_test.Test, newFormat.histoutilsex_test.Test, newFormat.incidentsvc_test.Test, newFormat.init_loop_check_off_test.Test, newFormat.init_loop_check_on_test.Test, newFormat.maps_test.Test, newFormat.ntuples_test.Test, newFormat.particlepropsvc_test.Test, newFormat.python_bin_module_test.Test, newFormat.qotdalg_test.Test, newFormat.randomnumber_test.Test, newFormat.selections_test.Test, newFormat.signal_handler_test.Test, newFormat.statsvcalg_test.Test, newFormat.statuscodecsvc_test.Test, newFormat.stringkey_test.Test, newFormat.templatedalg_pyopts_test.Test, newFormat.templatedalg_test.Test, newFormat.thistread_test.Test, newFormat.thistwrite_test.Test, newFormat.timing_histos_test.Test, newFormat.timing_test.Test, newFormat.tupleex1_test.Test, newFormat.tupleex2_test.Test, newFormat.tupleex3_test.Test, newFormat.tupleex4_test.Test, newFormat.tupleex_py_test.Test, newFormat.tupleex_pyopts_test.Test, newFormat.tupleex_test.Test, and QMTTest.QMTTest.

Public Member Functions

def __init__
 
def validator
 
def runTest
 
def ValidateOutput
 
def findReferenceBlock
 
def countErrorLines
 
def CheckTTreesSummaries
 
def CheckHistosSummaries
 
def validateWithReference
 

Public Attributes

 program
 
 args
 
 reference
 
 error_reference
 
 options
 
 stderr
 
 timeout
 
 exit_code
 
 environment
 
 target
 
 traceback
 
 unsupported_platforms
 
 signal
 
 status
 
 name
 
 causes
 
 timeOut
 
 result
 
 returnedCode
 
 out
 
 err
 
 proc
 

Detailed Description

Definition at line 17 of file BaseTest.py.

Constructor & Destructor Documentation

def BaseTest.BaseTest.__init__ (   self)

Definition at line 18 of file BaseTest.py.

18 
19  def __init__(self):
20  self.program=''
21  self.args=[]
22  self.reference=''
23  self.error_reference=''
24  self.options=''
25  self.stderr=''
26  self.timeout=120
27  self.exit_code=None
28  self.environment=None
29  self.target='local'
30  self.traceback=True
31  self.unsupported_platforms=[]
32  self.signal=None
33  #Variables not for users
34  self.status=None
35  self.name=''
36  self.causes = []
37  self.timeOut=False
38  self.result = Result(self)
39  self.returnedCode=0
40  self.out = ''
41  self.err = ''
42  self.proc=None

Member Function Documentation

def BaseTest.BaseTest.CheckHistosSummaries (   self,
  stdout = None,
  result = None,
  causes = None,
  dict = None,
  ignore = None 
) 
    Compare the TTree summaries in stdout with the ones in trees_dict or in
    the reference file. By default ignore the size, compression and basket
    fields.
    The presence of TTree summaries when none is expected is not a failure.

Definition at line 294 of file BaseTest.py.

294 
295  ignore = None):
296  """
297  Compare the TTree summaries in stdout with the ones in trees_dict or in
298  the reference file. By default ignore the size, compression and basket
299  fields.
300  The presence of TTree summaries when none is expected is not a failure.
301  """
302  if stdout is None : stdout=self.out
303  if result is None : result=self.result
304  if causes is None : causes=self.causes
305 
306  if dict is None:
307  lreference = _expandReferenceFileName(self,self.reference)
308  # call the validator if the file exists
309  if lreference and os.path.isfile(lreference):
310  dict = findHistosSummaries(open(lreference).read())
311  else:
312  dict = {}
313 
314  from pprint import PrettyPrinter
315  pp = PrettyPrinter()
316  if dict:
317  result["GaudiTest.Histos.expected"] = result.Quote(pp.pformat(dict))
318  if ignore:
319  result["GaudiTest.Histos.ignore"] = result.Quote(ignore)
320 
321  histos = findHistosSummaries(stdout)
322  failed = cmpTreesDicts(dict, histos, ignore)
323  if failed:
324  causes.append("histos summaries")
325  msg = "%s: %s != %s" % getCmpFailingValues(dict, histos, failed)
326  result["GaudiTest.Histos.failure_on"] = result.Quote(msg)
327  result["GaudiTest.Histos.found"] = result.Quote(pp.pformat(histos))
328 
329  return causes
BaseTest._expandReferenceFileName
def _expandReferenceFileName
Definition: BaseTest.py:999
BaseTest.getCmpFailingValues
def getCmpFailingValues
Definition: BaseTest.py:816
BaseTest.cmpTreesDicts
def cmpTreesDicts
Definition: BaseTest.py:785
BaseTest.findHistosSummaries
def findHistosSummaries
Definition: BaseTest.py:944
def BaseTest.BaseTest.CheckTTreesSummaries (   self,
  stdout = None,
  result = None,
  causes = None,
  trees_dict = None,
  ignore = r"Basket|.*size|Compression" 
) 
    Compare the TTree summaries in stdout with the ones in trees_dict or in
    the reference file. By default ignore the size, compression and basket
    fields.
    The presence of TTree summaries when none is expected is not a failure.

Definition at line 257 of file BaseTest.py.

257 
258  ignore = r"Basket|.*size|Compression"):
259  """
260  Compare the TTree summaries in stdout with the ones in trees_dict or in
261  the reference file. By default ignore the size, compression and basket
262  fields.
263  The presence of TTree summaries when none is expected is not a failure.
264  """
265  if stdout is None : stdout=self.out
266  if result is None : result=self.result
267  if causes is None : causes=self.causes
268  if trees_dict is None:
269  lreference = _expandReferenceFileName(self,self.reference)
270  # call the validator if the file exists
271  if lreference and os.path.isfile(lreference):
272  trees_dict = findTTreeSummaries(open(lreference).read())
273  else:
274  trees_dict = {}
275 
276  from pprint import PrettyPrinter
277  pp = PrettyPrinter()
278  if trees_dict:
279  result["GaudiTest.TTrees.expected"] = result.Quote(pp.pformat(trees_dict))
280  if ignore:
281  result["GaudiTest.TTrees.ignore"] = result.Quote(ignore)
282 
283  trees = findTTreeSummaries(stdout)
284  failed = cmpTreesDicts(trees_dict, trees, ignore)
285  if failed:
286  causes.append("trees summaries")
287  msg = "%s: %s != %s" % getCmpFailingValues(trees_dict, trees, failed)
288  result["GaudiTest.TTrees.failure_on"] = result.Quote(msg)
289  result["GaudiTest.TTrees.found"] = result.Quote(pp.pformat(trees))
290 
291  return causes
BaseTest.findTTreeSummaries
def findTTreeSummaries
Definition: BaseTest.py:764
BaseTest._expandReferenceFileName
def _expandReferenceFileName
Definition: BaseTest.py:999
BaseTest.getCmpFailingValues
def getCmpFailingValues
Definition: BaseTest.py:816
BaseTest.cmpTreesDicts
def cmpTreesDicts
Definition: BaseTest.py:785
def BaseTest.BaseTest.countErrorLines (   self,
  expected = {'ERROR':0,
  FATAL 
)

Definition at line 219 of file BaseTest.py.

219 
220  def countErrorLines(self, expected = {'ERROR':0, 'FATAL':0}, stdout=None, result=None,causes=None):
221  """
222  Count the number of messages with required severity (by default ERROR and FATAL)
223  and check if their numbers match the expected ones (0 by default).
224  The dictionary "expected" can be used to tune the number of errors and fatals
225  allowed, or to limit the number of expected warnings etc.
226  """
227 
228  if stdout is None : stdout=self.out
229  if result is None : result=self.result
230  if causes is None : causes=self.causes
231 
232  # prepare the dictionary to record the extracted lines
233  errors = {}
234  for sev in expected:
235  errors[sev] = []
236 
237  outlines = stdout.splitlines()
238  from math import log10
239  fmt = "%%%dd - %%s" % (int(log10(len(outlines)+1)))
240 
241  linecount = 0
242  for l in outlines:
243  linecount += 1
244  words = l.split()
245  if len(words) >= 2 and words[1] in errors:
246  errors[words[1]].append(fmt%(linecount,l.rstrip()))
247 
248  for e in errors:
249  if len(errors[e]) != expected[e]:
250  causes.append('%s(%d)'%(e,len(errors[e])))
251  result["GaudiTest.lines.%s"%e] = result.Quote('\n'.join(errors[e]))
252  result["GaudiTest.lines.%s.expected#"%e] = result.Quote(str(expected[e]))
253 
254  return causes
def BaseTest.BaseTest.findReferenceBlock (   self,
  reference = None,
  stdout = None,
  result = None,
  causes = None,
  signature_offset = 0,
  signature = None,
  id = None 
) 
    Given a block of text, tries to find it in the output. The block had to be identified by a signature line. By default, the first line is used as signature, or the line pointed to by signature_offset. If signature_offset points outside the block, a signature line can be passed as signature argument. Note: if 'signature' is None (the default), a negative signature_offset is interpreted as index in a list (e.g. -1 means the last line), otherwise the it is interpreted as the number of lines before the first one of the block the signature must appear. The parameter 'id' allow to distinguish between different calls to this function in the same validation code.

Definition at line 178 of file BaseTest.py.

178 
179  def findReferenceBlock(self,reference=None, stdout=None, result=None, causes=None, signature_offset=0, signature=None, id = None):
180  """
181  Given a block of text, tries to find it in the output. The block had to be identified by a signature line. By default, the first line is used as signature, or the line pointed to by signature_offset. If signature_offset points outside the block, a signature line can be passed as signature argument. Note: if 'signature' is None (the default), a negative signature_offset is interpreted as index in a list (e.g. -1 means the last line), otherwise the it is interpreted as the number of lines before the first one of the block the signature must appear. The parameter 'id' allow to distinguish between different calls to this function in the same validation code.
182  """
183 
184  if reference is None : reference=self.reference
185  if stdout is None : stdout=self.out
186  if result is None : result=self.result
187  if causes is None : causes=self.causes
188 
189  reflines = filter(None,map(lambda s: s.rstrip(), reference.splitlines()))
190  if not reflines:
191  raise RuntimeError("Empty (or null) reference")
192  # the same on standard output
193  outlines = filter(None,map(lambda s: s.rstrip(), stdout.splitlines()))
194 
195  res_field = "GaudiTest.RefBlock"
196  if id:
197  res_field += "_%s" % id
198 
199  if signature is None:
200  if signature_offset < 0:
201  signature_offset = len(reference)+signature_offset
202  signature = reflines[signature_offset]
203  # find the reference block in the output file
204  try:
205  pos = outlines.index(signature)
206  outlines = outlines[pos-signature_offset:pos+len(reflines)-signature_offset]
207  if reflines != outlines:
208  msg = "standard output"
209  # I do not want 2 messages in causes if teh function is called twice
210  if not msg in causes:
211  causes.append(msg)
212  result[res_field + ".observed"] = result.Quote("\n".join(outlines))
213  except ValueError:
214  causes.append("missing signature")
215  result[res_field + ".signature"] = result.Quote(signature)
216  if len(reflines) > 1 or signature != reflines[0]:
217  result[res_field + ".expected"] = result.Quote("\n".join(reflines))
218  return causes
Containers::map
struct GAUDI_API map
Parametrisation class for map-like implementation.
Definition: KeyedObjectManager.h:26
BaseTest.BaseTest.findReferenceBlock
def findReferenceBlock
Definition: BaseTest.py:178
def BaseTest.BaseTest.runTest (   self)

Definition at line 46 of file BaseTest.py.

46 
47  def runTest (self):
48  #Setting time
49  begining = int(time.time())
50  DOB = time.localtime()
51  dateOfBegining = str(DOB[1])+"/"+str(DOB[2])+"/"+str(DOB[0])+" "+str(DOB[3])+":"+str(DOB[4])+":"+str(DOB[5])
52 
53 
54  if self.options != '' :
55  if self.options.startswith("import") or self.options.startswith("from") or self.options.startswith("\nfrom") or self.options.startswith("\nimport") or (self.options.startswith("\n#") and not self.options.startswith('\n#include')):
56  optionFile = tempfile.NamedTemporaryFile(suffix='.py')
57  else :
58  optionFile = tempfile.NamedTemporaryFile(suffix='.opts')
59  optionFile.file.write(self.options)
60  optionFile.seek(0)
61  self.args.append(RationalizePath(optionFile.name))
62 
63  #If not specified, setting the environment
64  if self.environment is None : self.environment = os.environ
65  else : self.environment=dict(self.environment.items()+os.environ.items())
66 
67  #launching test in a different thread to handle timeout exception
68  def run (self=self):
69  def target (self=self) :
70  prog=''
71  if self.program != '' :
72  prog = RationalizePath(self.program)
73  elif "GAUDIEXE" in os.environ :
74  prog = os.environ["GAUDIEXE"]
75  else :
76  prog = "Gaudi.exe"
77 
78  dummy, prog_ext = os.path.splitext(prog)
79  if prog_ext not in [ ".exe", ".py", ".bat" ]:
80  prog += ".exe"
81  prog_ext = ".exe"
82 
83  prog = which(prog) or prog
84 
85  if prog_ext == ".py" :
86  params = ["/afs/cern.ch/user/v/valentin/workspace/Gaudi/build.x86_64-slc6-gcc48-opt/run","/afs/cern.ch/sw/lcg/external/Python/2.7.3/x86_64-slc6-gcc48-opt/bin/python",RationalizePath(prog)]+self.args
87  else :
88  params = ["/afs/cern.ch/user/v/valentin/workspace/Gaudi/build.x86_64-slc6-gcc48-opt/run", RationalizePath(prog)]+self.args
89 
90  print self.name
91  print params
92  self.proc= Popen(params,stdout=PIPE,stderr=PIPE, env=self.environment)
93  self.proc.wait()
94 
95  thread = threading.Thread(target=target)
96  thread.start()
97  #catching timeout
98  thread.join(self.timeout)
99 
100  if thread.is_alive():
101  self.proc.send_signal(signal.SIGABRT)
102  print "abortion"
103  thread.join()
104  self.timeOut = True
105  self.out = self.proc.stdout.read()
106  if self.traceback:
107  self.err = self.proc.stderr.read()
108  #Getting the error code
109  self.returnedCode = self.proc.returncode
110 
111  unsupPlat = True
112  for p in self.unsupported_platforms :
113  if re.search(p,platform.platform()):
114  unsupPlat=False
115 
116  if unsupPlat :
117  run()
118 
119  #End time
120  end=time.time()
121  #Time lasted
122  lasted = end-begining
123  #Getting date and local time
124  DOE = time.localtime()
125  dateOfEnding = str(DOE[1])+"/"+str(DOE[2])+"/"+str(DOE[0])+" "+str(DOE[3])+":"+str(DOE[4])+":"+str(DOE[5])
126 
127  if unsupPlat :
128  validatorRes = Result({'CAUSE':None,'EXCEPTION':None,'RESOURCE':None,'TARGET':None,'TRACEBACK':None,'START_TIME':None,'END_TIME':None,'TIMEOUT_DETAIL':None})
129  self.result=validatorRes
130 
131  if self.timeOut:
132  self.result["Abort cause"]=self.result.Quote("Event Timeout")
133 
134  self.result,self.causes=self.ValidateOutput(stdout=self.out,stderr=self.err,result=validatorRes)
135 
136  #Setting status
137  self.status = "Failed"
138  if self.signal is not None :
139  if (int(self.returnedCode) - int(self.signal) - 128)!=0:
140  self.causes.append('wrong return code')
141  if self.exit_code is not None:
142  if int(self.returnedCode) != int(self.exit_code) :
143  self.causes.append('wrong return code')
144  if self.returnedCode!=0 and self.exit_code is None and self.signal is None:
145  self.causes.append("Return code !=0")
146  if self.causes == []:
147  self.status = "Passed"
148  else :
149  self.status="SKIPPED"
150 
151  resultDic = {'Execution Time':lasted, 'Exit code':self.returnedCode, 'Start Time':dateOfBegining, 'End Time':dateOfEnding, 'Stderr':self.err,'Arguments':self.args, 'Environment':self.environment, 'Expected stderr':self.stderr, 'Status':self.status, 'Measurement':self.out, 'Program Name':self.program,'Name':self.name, 'Validator':self.validator,'Reference file':self.reference,'Error reference file':self.error_reference,'Causes':self.causes,'Validator results':self.result.annotations,'Unsupported platforms':self.unsupported_platforms}
152  return resultDic
153 
BaseTest.which
def which
Definition: BaseTest.py:436
BaseTest.RationalizePath
def RationalizePath
Definition: BaseTest.py:425
def BaseTest.BaseTest.ValidateOutput (   self,
  stdout,
  stderr,
  result 
)

Definition at line 158 of file BaseTest.py.

158 
159  def ValidateOutput(self, stdout, stderr, result):
160  causes = self.causes
161  reference =self.reference
162  error_reference=self.error_reference
163  #checking if default validation or not
164  if inspect.getsource(self.validator)!=""" def validator(self, stdout='',stderr=''):
165  pass
166 """ :
167  self.validator(stdout, stderr, result, causes, reference, error_reference)
168  else:
169  if self.stderr=='':
170  self.validateWithReference(stdout, stderr, result, causes)
171  elif stderr!=self.stderr:
172  self.causes.append("DIFFERENT STDERR THAN EXPECTED")
173 
174 
175  return result,causes
176 
177 
BaseTest.BaseTest.validateWithReference
def validateWithReference
Definition: BaseTest.py:330
def BaseTest.BaseTest.validateWithReference (   self,
  stdout = None,
  stderr = None,
  result = None,
  causes = None,
  preproc = None 
) 
    Default validation action: compare standard output and error to the
    reference files.

Definition at line 330 of file BaseTest.py.

330 
331  def validateWithReference(self, stdout=None, stderr=None, result=None, causes=None, preproc = None):
332  """
333  Default validation action: compare standard output and error to the
334  reference files.
335  """
336 
337  if stdout is None : stdout=self.out
338  if stderr is None : stderr=self.err
339  if result is None : result=self.result
340  if causes is None : causes=self.causes
341 
342  # set the default output preprocessor
343  if preproc is None:
344  preproc = normalizeExamples
345  # check standard output
346  lreference = _expandReferenceFileName(self,self.reference)
347  # call the validator if the file exists
348  if lreference and os.path.isfile(lreference):
349  result["GaudiTest.output_reference"] = lreference
350  causes += ReferenceFileValidator(lreference,
351  "standard output",
352  "GaudiTest.output_diff",
353  preproc = preproc)(stdout, result)
354  # Compare TTree summaries
355  causes = self.CheckTTreesSummaries(stdout, result, causes)
356  causes = self.CheckHistosSummaries(stdout, result, causes)
357  if causes: # Write a new reference file for stdout
358  try:
359  newref = open(lreference + ".new","w")
360  # sanitize newlines
361  for l in stdout.splitlines():
362  newref.write(l.rstrip() + '\n')
363  del newref # flush and close
364  except IOError:
365  # Ignore IO errors when trying to update reference files
366  # because we may be in a read-only filesystem
367  pass
368 
369  # check standard error
370  lreference = _expandReferenceFileName(self,self.error_reference)
371  # call the validator if we have a file to use
372  if lreference and os.path.isfile(self.error_reference):
373  result["GaudiTest.error_reference"] = self.error_reference
374  newcauses = ReferenceFileValidator(lreference, "standard error", "GaudiTest.error_diff", preproc = preproc)(stderr, result)
375  causes += newcauses
376  if newcauses: # Write a new reference file for stdedd
377  newref = open(self.reference + ".new","w")
378  # sanitize newlines
379  for l in stderr.splitlines():
380  newref.write(l.rstrip() + '\n')
381  del newref # flush and close
382  else:
383  causes += BasicOutputValidator(lreference, "standard error", "ExecTest.expected_stderr")(stderr, result)
384  return causes
385 
386 #---------------------------------------------------------------------------------------------------#
387 #---------------------------------------------------------------------------------------------------#
388 #-----------------------------------------GAUDI TOOLS-----------------------------------------------------#
389 #---------------------------------------------------------------------------------------------------#
390 #---------------------------------------------------------------------------------------------------#
BaseTest.BaseTest.validateWithReference
def validateWithReference
Definition: BaseTest.py:330
BaseTest._expandReferenceFileName
def _expandReferenceFileName
Definition: BaseTest.py:999
BaseTest.BaseTest.CheckTTreesSummaries
def CheckTTreesSummaries
Definition: BaseTest.py:257
BaseTest.BaseTest.CheckHistosSummaries
def CheckHistosSummaries
Definition: BaseTest.py:294
def BaseTest.BaseTest.validator (   self,
  stdout = '',
  stderr = '' 
)

Definition at line 43 of file BaseTest.py.

43 
44  def validator(self, stdout='',stderr=''):
45  pass

Member Data Documentation

BaseTest.BaseTest.args

Definition at line 20 of file BaseTest.py.

BaseTest.BaseTest.causes

Definition at line 35 of file BaseTest.py.

BaseTest.BaseTest.environment

Definition at line 27 of file BaseTest.py.

BaseTest.BaseTest.err

Definition at line 40 of file BaseTest.py.

BaseTest.BaseTest.error_reference

Definition at line 22 of file BaseTest.py.

BaseTest.BaseTest.exit_code

Definition at line 26 of file BaseTest.py.

BaseTest.BaseTest.name

Definition at line 34 of file BaseTest.py.

BaseTest.BaseTest.options

Definition at line 23 of file BaseTest.py.

BaseTest.BaseTest.out

Definition at line 39 of file BaseTest.py.

BaseTest.BaseTest.proc

Definition at line 41 of file BaseTest.py.

BaseTest.BaseTest.program

Definition at line 19 of file BaseTest.py.

BaseTest.BaseTest.reference

Definition at line 21 of file BaseTest.py.

BaseTest.BaseTest.result

Definition at line 37 of file BaseTest.py.

BaseTest.BaseTest.returnedCode

Definition at line 38 of file BaseTest.py.

BaseTest.BaseTest.signal

Definition at line 31 of file BaseTest.py.

BaseTest.BaseTest.status

Definition at line 33 of file BaseTest.py.

BaseTest.BaseTest.stderr

Definition at line 24 of file BaseTest.py.

BaseTest.BaseTest.target

Definition at line 28 of file BaseTest.py.

BaseTest.BaseTest.timeout

Definition at line 25 of file BaseTest.py.

BaseTest.BaseTest.timeOut

Definition at line 36 of file BaseTest.py.

BaseTest.BaseTest.traceback

Definition at line 29 of file BaseTest.py.

BaseTest.BaseTest.unsupported_platforms

Definition at line 30 of file BaseTest.py.

The documentation for this class was generated from the following file: