### Home Classes Methods

### In Files

- lib/taskjuggler/TextParser/Scanner.rb

### Parent

Object

### Namespace

- CLASS TaskJuggler::TextParser::Scanner::BufferStreamHandle
- CLASS TaskJuggler::TextParser::Scanner::FileStreamHandle
- CLASS TaskJuggler::TextParser::Scanner::MacroStackEntry
- CLASS TaskJuggler::TextParser::Scanner::StreamHandle

### Methods

- ::new
- #addMacro
- #addPattern
- #close
- #error
- #expandMacro
- #fileName
- #include
- #macroDefined?
- #mode=
- #nextToken
- #open
- #returnToken
- #sourceFileInfo
- #warning

### Files

- CHANGELOG
- COPYING
- README.rdoc

### Class/Module Index

Quicksearch

- TaskJuggler
- TaskJuggler::Account
- TaskJuggler::AccountAttribute
- TaskJuggler::AccountCredit
- TaskJuggler::AccountCreditListAttribute
- TaskJuggler::AccountListRE
- TaskJuggler::AccountScenario
- TaskJuggler::AlertLevelDefinition
- TaskJuggler::AlertLevelDefinitions
- TaskJuggler::Allocation
- TaskJuggler::AllocationAttribute
- TaskJuggler::AttributeBase
- TaskJuggler::AttributeDefinition
- TaskJuggler::AttributeOverwrite
- TaskJuggler::BatchProcessor
- TaskJuggler::Booking
- TaskJuggler::BookingListAttribute
- TaskJuggler::BooleanAttribute
- TaskJuggler::CSVFile
- TaskJuggler::CellSettingPattern
- TaskJuggler::CellSettingPatternList
- TaskJuggler::Charge
- TaskJuggler::ChargeListAttribute
- TaskJuggler::ChargeSet
- TaskJuggler::ChargeSetListAttribute
- TaskJuggler::ChartPlotter
- TaskJuggler::ChartPlotterError
- TaskJuggler::CollisionDetector
- TaskJuggler::ColumnListAttribute
- TaskJuggler::ColumnTable
- TaskJuggler::Daemon
- TaskJuggler::DaemonConnector
- TaskJuggler::DaemonConnectorMixin
- TaskJuggler::DataCache
- TaskJuggler::DataCacheEntry
- TaskJuggler::DateAttribute
- TaskJuggler::DefinitionListAttribute
- TaskJuggler::DependencyListAttribute
- TaskJuggler::DurationAttribute
- TaskJuggler::ExportRE
- TaskJuggler::FileList
- TaskJuggler::FileRecord
- TaskJuggler::FixnumAttribute
- TaskJuggler::FlagListAttribute
- TaskJuggler::FloatAttribute
- TaskJuggler::FormatListAttribute
- TaskJuggler::GanttChart
- TaskJuggler::GanttContainer
- TaskJuggler::GanttHeader
- TaskJuggler::GanttHeaderScaleItem
- TaskJuggler::GanttLine
- TaskJuggler::GanttLoadStack
- TaskJuggler::GanttMilestone
- TaskJuggler::GanttRouter
- TaskJuggler::GanttTaskBar
- TaskJuggler::HTMLDocument
- TaskJuggler::HTMLElements
- TaskJuggler::HTMLGraphics
- TaskJuggler::ICalReport
- TaskJuggler::ICalendar
- TaskJuggler::ICalendar::Component
- TaskJuggler::ICalendar::Event
- TaskJuggler::ICalendar::Journal
- TaskJuggler::ICalendar::Person
- TaskJuggler::ICalendar::Todo
- TaskJuggler::Interval
- TaskJuggler::IntervalList
- TaskJuggler::JobInfo
- TaskJuggler::Journal
- TaskJuggler::JournalEntry
- TaskJuggler::JournalEntryList
- TaskJuggler::JournalSortListAttribute
- TaskJuggler::KateSyntax
- TaskJuggler::KeywordArray
- TaskJuggler::KeywordDocumentation
- TaskJuggler::Leave
- TaskJuggler::LeaveAllowance
- TaskJuggler::LeaveAllowanceList
- TaskJuggler::LeaveAllowanceListAttribute
- TaskJuggler::LeaveList
- TaskJuggler::LeaveListAttribute
- TaskJuggler::Limits
- TaskJuggler::Limits::Limit
- TaskJuggler::LimitsAttribute
- TaskJuggler::ListAttributeBase
- TaskJuggler::Log
- TaskJuggler::LogicalAttribute
- TaskJuggler::LogicalExpression
- TaskJuggler::LogicalExpressionAttribute
- TaskJuggler::LogicalExpressionListAttribute
- TaskJuggler::LogicalFlag
- TaskJuggler::LogicalFunction
- TaskJuggler::LogicalOperation
- TaskJuggler::ManagerResponsibilities
- TaskJuggler::ManagerStatusRecord
- TaskJuggler::Message
- TaskJuggler::MessageHandler
- TaskJuggler::MessageHandlerInstance
- TaskJuggler::MspXmlRE
- TaskJuggler::Navigator
- TaskJuggler::NavigatorElement
- TaskJuggler::NikuProject
- TaskJuggler::NikuReport
- TaskJuggler::NikuResource
- TaskJuggler::NodeListAttribute
- TaskJuggler::PTNProxy
- TaskJuggler::Painter
- TaskJuggler::Painter::Circle
- TaskJuggler::Painter::Color
- TaskJuggler::Painter::Element
- TaskJuggler::Painter::Ellipse
- TaskJuggler::Painter::FontMetrics
- TaskJuggler::Painter::FontMetricsData
- TaskJuggler::Painter::Group
- TaskJuggler::Painter::Line
- TaskJuggler::Painter::Points
- TaskJuggler::Painter::PolyLine
- TaskJuggler::Painter::Primitives
- TaskJuggler::Painter::Rect
- TaskJuggler::Painter::SVGSupport
- TaskJuggler::Painter::Text
- TaskJuggler::PlaceHolderCell
- TaskJuggler::ProcessIntercom
- TaskJuggler::ProcessIntercomIface
- TaskJuggler::Project
- TaskJuggler::ProjectBroker
- TaskJuggler::ProjectBrokerIface
- TaskJuggler::ProjectFileParser
- TaskJuggler::ProjectFileScanner
- TaskJuggler::ProjectRecord
- TaskJuggler::ProjectServer
- TaskJuggler::ProjectServerIface
- TaskJuggler::PropertyAttribute
- TaskJuggler::PropertyList
- TaskJuggler::PropertySet
- TaskJuggler::PropertyTreeNode
- TaskJuggler::Query
- TaskJuggler::RTFHandlers
- TaskJuggler::RTFNavigator
- TaskJuggler::RTFQuery
- TaskJuggler::RTFReport
- TaskJuggler::RTFReportLink
- TaskJuggler::RTFWithQuerySupport
- TaskJuggler::RealFormat
- TaskJuggler::RealFormatAttribute
- TaskJuggler::ReferenceAttribute
- TaskJuggler::Report
- TaskJuggler::ReportBase
- TaskJuggler::ReportContext
- TaskJuggler::ReportScenario
- TaskJuggler::ReportServer
- TaskJuggler::ReportServerIface
- TaskJuggler::ReportServerRecord
- TaskJuggler::ReportServlet
- TaskJuggler::ReportTable
- TaskJuggler::ReportTableCell
- TaskJuggler::ReportTableColumn
- TaskJuggler::ReportTableLegend
- TaskJuggler::ReportTableLine
- TaskJuggler::Resource
- TaskJuggler::ResourceListAttribute
- TaskJuggler::ResourceListRE
- TaskJuggler::ResourceScenario
- TaskJuggler::RichText
- TaskJuggler::RichTextAttribute
- TaskJuggler::RichTextDocument
- TaskJuggler::RichTextElement
- TaskJuggler::RichTextFunctionExample
- TaskJuggler::RichTextFunctionHandler
- TaskJuggler::RichTextImage
- TaskJuggler::RichTextIntermediate
- TaskJuggler::RichTextParser
- TaskJuggler::RichTextScanner
- TaskJuggler::RichTextSnip
- TaskJuggler::RichTextSyntaxRules
- TaskJuggler::Scenario
- TaskJuggler::ScenarioData
- TaskJuggler::ScenarioListAttribute
- TaskJuggler::Scoreboard
- TaskJuggler::ScoreboardInterval
- TaskJuggler::SheetHandlerBase
- TaskJuggler::SheetReceiver
- TaskJuggler::SheetSender
- TaskJuggler::Shift
- TaskJuggler::ShiftAssignment
- TaskJuggler::ShiftAssignments
- TaskJuggler::ShiftAssignmentsAttribute
- TaskJuggler::ShiftScenario
- TaskJuggler::SimpleQueryExpander
- TaskJuggler::SortListAttribute
- TaskJuggler::StatusSheetReceiver
- TaskJuggler::StatusSheetReport
- TaskJuggler::StatusSheetSender
- TaskJuggler::StdIoWrapper
- TaskJuggler::StringAttribute
- TaskJuggler::SymbolAttribute
- TaskJuggler::SymbolListAttribute
- TaskJuggler::SyntaxReference
- TaskJuggler::TOCEntry
- TaskJuggler::TSResourceRecord
- TaskJuggler::TSTaskRecord
- TaskJuggler::TableColumnDefinition
- TaskJuggler::TableColumnSorter
- TaskJuggler::TableOfContents
- TaskJuggler::TableReport
- TaskJuggler::TableReportColumn
- TaskJuggler::TagFile
- TaskJuggler::TagFile::TagFileEntry
- TaskJuggler::Task
- TaskJuggler::TaskDepListAttribute
- TaskJuggler::TaskDependency
- TaskJuggler::TaskListAttribute
- TaskJuggler::TaskListRE
- TaskJuggler::TaskScenario
- TaskJuggler::TernarySearchTree
- TaskJuggler::TextFormatter
- TaskJuggler::TextParser
- TaskJuggler::TextParser::Macro
- TaskJuggler::TextParser::MacroTable
- TaskJuggler::TextParser::Pattern
- TaskJuggler::TextParser::Rule
- TaskJuggler::TextParser::Scanner
- TaskJuggler::TextParser::Scanner::BufferStreamHandle
- TaskJuggler::TextParser::Scanner::FileStreamHandle
- TaskJuggler::TextParser::Scanner::MacroStackEntry
- TaskJuggler::TextParser::Scanner::StreamHandle
- TaskJuggler::TextParser::SourceFileInfo
- TaskJuggler::TextParser::StackElement
- TaskJuggler::TextParser::State
- TaskJuggler::TextParser::StateTransition
- TaskJuggler::TextParser::TextParserResultArray
- TaskJuggler::TextParser::TokenDoc
- TaskJuggler::TextReport
- TaskJuggler::TimeInterval
- TaskJuggler::TimeIntervalListAttribute
- TaskJuggler::TimeSheet
- TaskJuggler::TimeSheetReceiver
- TaskJuggler::TimeSheetRecord
- TaskJuggler::TimeSheetReport
- TaskJuggler::TimeSheetSender
- TaskJuggler::TimeSheetSummary
- TaskJuggler::TimeSheets
- TaskJuggler::Tj3
- TaskJuggler::Tj3AppBase
- TaskJuggler::Tj3Client
- TaskJuggler::Tj3Daemon
- TaskJuggler::Tj3Man
- TaskJuggler::Tj3SheetAppBase
- TaskJuggler::Tj3SsReceiver
- TaskJuggler::Tj3SsSender
- TaskJuggler::Tj3TsReceiver
- TaskJuggler::Tj3TsSender
- TaskJuggler::Tj3TsSummary
- TaskJuggler::Tj3WebD
- TaskJuggler::TjException
- TaskJuggler::TjRuntimeError
- TaskJuggler::TjTime
- TaskJuggler::TjpExample
- TaskJuggler::TjpExportRE
- TaskJuggler::TjpSyntaxRules
- TaskJuggler::TraceReport
- TaskJuggler::URLParameter
- TaskJuggler::UserManual
- TaskJuggler::VimSyntax
- TaskJuggler::WebServer
- TaskJuggler::WelcomePage
- TaskJuggler::WorkingHours
- TaskJuggler::WorkingHoursAttribute
- TaskJuggler::XMLBlob
- TaskJuggler::XMLComment
- TaskJuggler::XMLDocument
- TaskJuggler::XMLElement
- TaskJuggler::XMLNamedText
- TaskJuggler::XMLText
- Diff
- Diff::Hunk
- AppConfig
- Diffable
- DiffableString
- Object
- RuntimeConfig
- String

No matching classes.

# TaskJuggler::TextParser::Scanner

The Scanner class is an abstract text scanner with support for nested include files and text macros. The tokenizer will operate on rules that must be provided by a derived class. The scanner is modal. Each mode operates only with the subset of token patterns that are assigned to the current mode. The current line is tracked accurately and can be used for error reporting. The scanner can operate on Strings or Files.

### Public Class Methods

new(masterFile, log, tokenPatterns, defaultMode) click to toggle source

Create a new instance of Scanner. masterFile must be a String that either contains the name of the file to start with or the text itself. messageHandler is a MessageHandler that is used for error messages. log is a Log to report progress and status.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 207
    def initialize(masterFile, log, tokenPatterns, defaultMode)
      @masterFile = masterFile
      @messageHandler = TaskJuggler::MessageHandlerInstance.instance
      @log = log
      # This table contains all macros that may be expanded when found in the
      # text.
      @macroTable = MacroTable.new
      # The currently processed IO object.
      @cf = nil
      # This Array stores the currently processed nested files. It's an Array
      # of Arrays. The nested Array consists of 2 elements, the IO object and
      # the @tokenBuffer.
      @fileStack = []
      # This flag is set if we have reached the end of a file. Since we will
      # only know when the next new token is requested that the file is really
      # done now, we have to use this flag.
      @finishLastFile = false
      # True if the scanner operates on a buffer.
      @fileNameIsBuffer = false
      # A SourceFileInfo of the start of the currently processed token.
      @startOfToken = nil
      # Line number correction for error messages.
      @lineDelta = 0
      # Lists of regexps that describe the detectable tokens. The Arrays are
      # grouped by mode.
      @patternsByMode = { }
      # The currently active scanner mode.
      @scannerMode = nil
      # The mode that the scanner is in at the start and end of file
      @defaultMode = defaultMode
      # Points to the currently active pattern set as defined by the mode.
      @activePatterns = nil

      tokenPatterns.each do |pat|
        type = pat[0]
        regExp = pat[1]
        mode = pat[2] || :tjp
        postProc = pat[3]
        addPattern(type, regExp, mode, postProc)
      end
      self.mode = defaultMode
    end

### Public Instance Methods

addMacro(macro) click to toggle source

Add a Macro to the macro translation table.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 433
    def addMacro(macro)
      @macroTable.add(macro)
    end

addPattern(type, regExp, mode, postProc = nil) click to toggle source

Add a new pattern to the scanner. type is either nil for tokens that will be ignored, or some identifier that will be returned with each token of this type. regExp is the RegExp that describes the token. mode identifies the scanner mode where the pattern is active. If it's only a single mode, mode specifies the mode directly. For multiple modes, it's an Array of modes. postProc is a method reference. This method is called after the token has been detected. The method gets the type and the matching String and returns them again in an Array.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 258
    def addPattern(type, regExp, mode, postProc = nil)
      if mode.is_a?(Array)
        mode.each do |m|
          # The pattern is active in multiple modes
          @patternsByMode[m] = [] unless @patternsByMode.include?(m)
          @patternsByMode[m] << [ type, regExp, postProc ]
        end
      else
        # The pattern is only active in one specific mode.
        @patternsByMode[mode] = [] unless @patternsByMode.include?(mode)
        @patternsByMode[mode] << [ type, regExp, postProc ]
      end
    end

close() click to toggle source

Finish processing and reset all data structures.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 302
    def close
      unless @fileNameIsBuffer
        @log.startProgressMeter("Reading file #{@masterFile}")
        @log.stopProgressMeter
      end
      @fileStack = []
      @cf = @tokenBuffer = nil
    end

error(id, text, sfi = nil, data = nil) click to toggle source

Call this function to report any errors related to the parsed input.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 464
    def error(id, text, sfi = nil, data = nil)
      message(:error, id, text, sfi, data)
    end

expandMacro(prefix, args, callLength) click to toggle source

Expand a macro and inject it into the input stream. prefix is any string that was found right before the macro call. We have to inject it before the expanded macro. args is an Array of Strings. The first is the macro name, the rest are the parameters. callLength is the number of characters for the complete macro call "${...}".

    # File lib/taskjuggler/TextParser/Scanner.rb, line 447
    def expandMacro(prefix, args, callLength)
      # Get the expanded macro from the @macroTable.
      macro, text = @macroTable.resolve(args, sourceFileInfo)

      # If the expanded macro is empty, we can ignore it.
      return if text == ''

      unless macro && text
        error('undefined_macro', "Undefined macro '#{args[0]}' called")
      end

      unless @cf.injectMacro(macro, args, prefix + text, callLength)
        error('macro_stack_overflow', "Too many nested macro calls.")
      end
    end

fileName() click to toggle source

Return the name of the currently processed file. If we are working on a text buffer, the text will be returned.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 359
    def fileName
      @cf ? @cf.fileName : @masterFile
    end

include(includeFileName, sfi, &block) click to toggle source

Continue processing with a new file specified by includeFileName. When this file is finished, we will continue in the old file after the location where we started with the new file. The method returns the full qualified name of the included file.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 315
    def include(includeFileName, sfi, &block)
      if includeFileName[0] != '/'
        pathOfCallingFile = @fileStack.last[0].dirname
        path = pathOfCallingFile.empty? ? '' : pathOfCallingFile + '/'
        # If the included file is not an absolute name, we interpret the file
        # name relative to the including file.
        includeFileName = path + includeFileName
      end

      # Try to dectect recursive inclusions. This will not work if files are
      # accessed via filesystem links.
      @fileStack.each do |entry|
        if includeFileName == entry[0].fileName
          error('include_recursion',
                "Recursive inclusion of #{includeFileName} detected", sfi)
        end
      end

      # Save @tokenBuffer in the record of the parent file.
      @fileStack.last[1] = @tokenBuffer unless @fileStack.empty?
      @tokenBuffer = nil
      @finishLastFile = false

      # Open the new file and push the handle on the @fileStack.
      begin
        @fileStack << [ (@cf = FileStreamHandle.new(includeFileName, @log,
                                                    self)), nil, block ]
        @log.msg { "Parsing file #{includeFileName}" }
      rescue StandardError
        error('bad_include', "Cannot open include file #{includeFileName}", sfi)
      end

      # Return the name of the included file.
      includeFileName
    end

macroDefined?(name) click to toggle source

Return true if the Macro name has been added already.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 438
    def macroDefined?(name)
      @macroTable.include?(name)
    end

mode=(newMode) click to toggle source

Switch the parser to another mode. The scanner will then only detect patterns of that newMode.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 274
    def mode=(newMode)
      #puts "**** New mode: #{newMode}"
      @activePatterns = @patternsByMode[newMode]
      raise "Undefined mode #{newMode}" unless @activePatterns
      @scannerMode = newMode
    end

nextToken() click to toggle source

Return the next token from the input stream. The result is an Array with 3 entries: the token type, the token String and the SourceFileInfo where the token started.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 378
    def nextToken
      # If we have a pushed-back token, return that first.
      unless @tokenBuffer.nil?
        res = @tokenBuffer
        @tokenBuffer = nil
        return res
      end

      if @finishLastFile
        # The previously processed file has now really been processed to
        # completion. Close it and remove the corresponding entry from the
        # @fileStack.
        @finishLastFile = false
        #@log.msg { "Completed file #{@cf.fileName}" }

        # If we have a block to be executed on EOF, we call it now.
        onEof = @fileStack.last[2]
        onEof.call if onEof

        @cf.close if @cf
        @fileStack.pop

        if @fileStack.empty?
          # We are done with the top-level file now.
          @cf = @tokenBuffer = nil
          @finishLastFile = true
          return [ :endOfText, '<EOT>', @startOfToken ]
        else
          # Continue parsing the file that included the current file.
          @cf, tokenBuffer = @fileStack.last
          @log.msg { "Parsing file #{@cf.fileName} ..." }
          # If we have a left over token from previously processing this file,
          # return it now.
          if tokenBuffer
            @finishLastFile = true if tokenBuffer[0] == :eof
            return tokenBuffer
          end
        end
      end

      scanToken
    end

open(fileNameIsBuffer = false) click to toggle source

Start the processing. if fileNameIsBuffer is true, we operate on a String, else on a File.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 284
    def open(fileNameIsBuffer = false)
      @fileNameIsBuffer = fileNameIsBuffer
      if fileNameIsBuffer
        @fileStack = [ [ @cf = BufferStreamHandle.new(@masterFile, @log, self),
                         nil, nil ] ]
      else
        begin
          @fileStack = [ [ @cf = FileStreamHandle.new(@masterFile, @log, self),
                           nil, nil ] ]
        rescue IOError, SystemCallError
          error('open_file', "Cannot open file #{@masterFile}: #{$!}")
        end
      end
      @masterPath = @cf.dirname + '/'
      @tokenBuffer = nil
    end

returnToken(token) click to toggle source

Return a token to retrieve it with the next nextToken() call again. Only 1 token can be returned before the next nextToken() call.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 423
    def returnToken(token)
      #@log.msg { "-> Returning Token: [#{token[0]}][#{token[1]}]" }
      unless @tokenBuffer.nil?
        $stderr.puts @tokenBuffer
        raise "Fatal Error: Cannot return more than 1 token in a row"
      end
      @tokenBuffer = token
    end

sourceFileInfo() click to toggle source

Return SourceFileInfo for the current processing prosition.

    # File lib/taskjuggler/TextParser/Scanner.rb, line 352
    def sourceFileInfo
      @cf ? SourceFileInfo.new(fileName, @cf.lineNo - @lineDelta, 0) :
            SourceFileInfo.new(@masterFile, 0, 0)
    end

warning(id, text, sfi = nil, data = nil) click to toggle source

    # File lib/taskjuggler/TextParser/Scanner.rb, line 468
    def warning(id, text, sfi = nil, data = nil)
      message(:warning, id, text, sfi, data)
    end

[Validate]

Generated with the Darkfish Rdoc Generator 2.