Welcome to TextPipe

TextPipe: Text Conversion, Reformatting and Extraction Workbench
 Welcome to TextPipe
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

For new users

Remember - just press F1 for help at any time, or choose Index or Contents from the Help Menu.

· TextPipe in a Nutshell
· What is TextPipe?
· What can I use TextPipe for?
· How do I buy TextPipe?
· Tutorial lessons, Quick Start guide, common tasks etc *Strongly recommended for beginners*
· Frequently Asked Question's and demonstrations (online)
· Discussion forums (online)

For experienced users

· An Introduction to Pattern Matching
· Restricting filters to only operate in part of a document

For advanced users

· Advanced Topics - automation and troubleshooting

For previous users

· What's new in this version?

Updates and discussions

We're always adding enhancements to TextPipe. If you'd like to be notified when enhancements are available, please subscribe to the TextPipe update list below. We also have a discussion group for TextPipe users - an interactive forum for questions and answers.

We also run an extensive discussion forum.

See also

Features

More Reviews (online)

Contact Us Support Community Tutorials and User Guides (online) Submit feedback on this topic
 Copyright © 1999-2014 DataMystic . All rights reserved.


TextPipe in a Nutshell

 TextPipe in a Nutshell
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe is a text workbench for converting, modifying and extracting text. Click here for Common tasks and how to perform them

To use TextPipe...

1. On the Filters to Apply tab, tell TextPipe what to do by choosing filters from the Convert, Add, Remove, Email, Replace, Special, Map and Restrict menus
2. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
3. Click the [Go] button.

Testing filters first...

You can test a filter by

· Pasting text into the Trial Run Area, then clicking the [Trial Run] button

OR

· Clicking the File Output filter at the base of the filter tree, ensure that Test Mode is checked, then click the [Go] button.

Designing filters...

TextPipe's filter list is built up from a series of modular filters, each performing a small task. For example, the task of extracting email addresses from a group of files might look like this:

1. Email menu\Extract email addresses - returns one email address per line
2. Special menu\Merge - merge all the email addresses into one big file
3. Special menu\Sort (case insensitive, removing duplicates)
4. Remove menu\Remove matching lines (to remove your own email address, or known bad email addresses)

To use these addresses in Outlook, you must convert them from one email address per line to a semi-colon separated list of email addresses.

1. Convert Menu\End of line characters (convert from DOS format to the string ';')

To design a filter, you may have to reduce the problem down to a series of simpler steps. Use Special menu\Comment filters to document your design.

More resources

· Tutorial lessons, Quick Start guide, common tasks etc *Strongly recommended for beginners*
· An Introduction to Pattern Matching
· Frequently Asked Questions and demonstrations (online)
· Restricting filters to only operate in part of a document
· Advanced Topics - automation and troubleshooting


What is TextPipe?

 What is TextPipe?
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe™ is the fastest and most powerful text manipulation software you'll ever use. With TextPipe's easy to use interface you can achieve in seconds what would take you hours with other software. No two text processing tasks are alike - and only TextPipe can provide the flexibility and incredible text manipulation capabilities to get the job done.

TextPipe is a text processing application that takes a group of files and applies a set of operations or filters to each file in turn. Each filter performs an operation such as a search and replace, adding text to the left margin, converting end of line characters etc, and then passes the result on to the next filter, just as though the entire file had been processed using that filter first.

TextPipe has an extensive range of over 100 filters for adding, deleting, replacing, sorting and transforming text. Common tasks like converting files between PC, Macintosh, Mainframe and Unix formats are well catered for. With TextPipe's command-line automation or COM interface, complex processing tasks can be performed without user intervention.

Filters included can also split and join files, add line numbers, word wrap, convert between OEM and ANSI and remove duplicate lines/HTML/columns/binary characters/ANSI codes.

Web sites can be maintained easily with multi-file search and replace, and the ability to add standard text (such as copyright messages or banner advertising) to the start and end of each file. Multiple spaces and blank lines can be quickly removed to improve download times.

TextPipe replaces a substantial set of smaller text utilities with a unified and easy-to-use GUI. General users, web authors, administrators and programmers can readily perform complex text-processing with a minimum of technical knowledge.

zd5star Ziff Davis rates it as a 5 Star Editor’s Pick - "Despite all these bells and whistles, TextPipe is amazingly easy to use."

See also

Features

New features in this release

How can I use TextPipe?

More Reviews


Lessons 1-7

 Lessons 1-7
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

We recommend that you print out each lesson to make it easier to follow while using the program. To print this page, click the Print button above (and again as you get to each new lesson).

These Tutorials give you step by step instructions to introduce you to TextPipe.

1. Lesson 1 - Basics
2. Lesson 2 - Changing the Filter List
3. Lesson 3 - Moving and deleting Filters
4. Lesson 4 - Adding files
5. Lesson 5 - Restrictions and sub filters
6. Lesson 6 - Removing text surrounded by other text
7. Lesson 7 - Reformatting text surrounded by other text

TextPipe for askSam

1. TextPipe for askSam tutorial (online)

Advanced Topics

1. Inserting data into a database
2. Web site mining, mainframe reports and more (online)

Back to User Assistance Page


Lesson 1 - Basics

 Lesson 1 - Basics
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

In this lesson you'll learn about:

· What is TextPipe?
· Opening TextPipe
· The main elements of TextPipe's screen
· How to use the Trial Run area
· How to open (load) and save filters.

What is TextPipe?

TextPipe is a powerful text transformation workbench that performs an enormous range of conversions and manipulations to text files. It is unique in the number of transformations that can be performed at one time, in the power and flexibility of its transformations, in the size of files it can handle and more. Data can be extracted from databases in a variety of text formats, and custom transformations can be written in VBScript, JScript and other scripting languages.

Opening TextPipe

If you haven't already started TextPipe, start it now by

1. Double-clicking on the TextPipe icon textpipe on the desktop OR
2. Clicking on TextPipe in the Start Menu\Programs\TextPipe folder

TextPipe starts with a new filter list (you may have to click Ok to clear the About dialog in the evaluation version, and close the Tip Of The Day and Filter Wizard dialogs).

The TextPipe Screen

At the very top of the TextPipe window is the Title Bar. The Title Bar tells you which filter list you are working on and whether it has been modified or not. Below the Title Bar is the Menu Bar, with the customary File, Edit, Window and Help menus, along with a Filter menu that categorize text processing filters into functional groups (we'll come back to filters in a minute).

Below the Menu Bar is the Tool Bar. The Tool Bar has icons for the most commonly used features of TextPipe, such as saving and loading filter list files, cutting and pasting and so on. The Tool Bar saves you having to navigate through the menus. If you hover the mouse cursor over a Tool Bar icon for a second, a hint will be displayed beside the mouse cursor that describes what the icon is for. At the same time, a helpful message will be displayed in the Status Bar at the base of TextPipe's window. You should watch for helpful text in the Status Bar whenever the mouse cursor is over a field in TextPipe's window, because the Status Bar always has extra information for you.

The Tab Bar is below the Tool Bar, and it contains a group of tabs that divide up the main tasks within TextPipe. We'll cover it in more detail below.

What is a filter list?

A TextPipe filter list consists of:

· A list of operations to perform (filters)
· A list of files that will be processed
· (Optionally) Trial text that can be used to test the filter list.

To understand this more fully, we'll open a sample filter.

Opening (loading) a filter list

1. On the File Menu, click Open
2. The Open Filter List window appears
3. Double click on the Tutorial folder (if you can't find it, it's usually in My Documents\DataMystic\TextPipe\Tutorial). Note - you can load any filter from the subject area subfolders such as coding, conversions, email, etext, fun, html, sysadmin or vbscript
4. Double click on 'lesson1'
5. TextPipe opens the lesson1 filter list.

The default tab is the Filters to Apply tab. It shows which filters will be performed, and in what order. Order is very important! In our lesson1 filter list, the only filter we will use is a straight-forward search and replace filter, which replaces capital T with lowercase T. We will cover the File Input and File Output filters later.

Click on the Files to Process tab. Although currently empty, this is where we list all the files that should be processed (using drag and drop, the [...] button, or the Folder button at the right.

Click on the Trial Run tab. The Trial Run Input area already has some sample text. When we test the filter with a Trial Run, the Trial Run output appears on the right hand side so that we can compare the output with the original, and make sure that the filter works as we expect. This can be really handy for complex filters.

Click the Trial Run button at the base of TextPipe's window. Some new text appears on the right hand side - with all capital T's replaced with lowercase t's (3 in all). This is exactly what we expect - because we have a single filter that is called Replace [T] with [t].

Add the text 'This is my line of text.' to the end of the text in the Trial Input area, and click the Trial Run button again. The new output text is shown.

Saving a filter list

If you want to save the filter list:

1. On the File Menu, click Save.

If you want to save the filter list with a new name:

1. On the File Menu, click Save As
2. The Save Filter List window appears
3. Type in a new name
4. Click the Save button.

In the next lesson we'll look at

· How to add filters to the filter list
· The importance of filter order.

Exercise 1

1. Open one of the filters in TextPipe's 'fun' folder - try Elmer Fudd.fll (usually C:\Program Files\TextPipe\Fun)
2. Type some text of your own into the Trial Run Input area
3. Click the Trial Run button to see what they do.

Tutorial Index


Lesson 2 - Changing the filter list

 Lesson 2 - Changing the filter list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

In the last lesson we covered

· The main elements of TextPipe's screen
· How to use the Trial Run area
· How to open and save filters.

In this lesson we'll look at

· How to add filters to the filter list
· The importance of filter order.

Before we begin, open the filter called 'tutorial\lesson2.fll'

Adding Filters to the Filter List

New filters can be added to the filter list to perform additional processing steps. Filters are listed in two places - in the Filter Menu (in the Menu Bar), and also in the the All Available Filters tree view (on the Filters to Apply tab), which is more convenient for adding multiple filters. Double click filters in the tree view to add them to My Filter List, or select them from the Filter Menu. It will be added below the currently selected filter.

Filters cannot be added or moved above the File input filter or below the File output filter. It doesn't make sense to put filters before or after these items.

To demonstrate adding new filters, we will add to the filter list we started with in Lesson 1. Let's add a simple search and replace filter - but this time, replacing capital 'T' and replacing with '*':

1. Select (click on the Replace filter (replacing 'T' with 't') in My Filter List
2. In All Available Filters, click on the [+] sign next to Replace, and then double-click Find exactly
3. A Replace filter is added below the current filter
4. Type the capital letter 'T' (without quotes) in the Find section
5. Type the letter '*' (without quotes) in the Replace With section
6. Ensure the Match Case check box is checked
7. Click the Trial Run button.

The text will be unchanged from before - the addition of the new filter has not made any difference.

Why? Order is important!

Filters are always applied from top to bottom, corresponding to the way they are shown.

The order that filters are applied is VERY important to TextPipe. Let's see why.

1. Switch back to the Filters to Apply tab (remember - you can click the Trial Run button to toggle back)
2. Select the Replace [T] with [*] filter
3. Click the Up arrow button at the bottom of the window, just to the right of the [Remove All] button
4. The Replace [T] with [*] filter is now moved up one position
5. Click the Trial Run button.

Now the changes are made correctly. The second filter is unnecessary because all capital T's have already been replaced.

An important part of advanced use of TextPipe is understanding how to break a task down into smaller steps. There are thousands of uses for TextPipe's filtering by just combining filters together.

In the next lesson we'll look at

· How to remove filters from the filter list
· How to move filters in the filter list
· How to disable filters.

Exercise 2

Position the mouse cursor over each icon in TextPipe's toolbar (at the top of TextPipe's window), and read the hint that appears in TextPipe's status bar (at the base of TextPipe's window).

Hints are also available on every control in TextPipe's windows.

Tutorial Index


Lesson 3 - Moving and deleting filters

 Lesson 3 - Moving and deleting filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

In the last lesson we covered

· How to add filters to the filter list
· The importance of filter order.

In this lesson we'll look at

· How to move filters in the filter list
· How to remove filters from the filter list
· How to disable filters.

Moving Filters in the Filter List

In the previous lesson we briefly touched on how to move filters in the filter list.

To move a filter up or down:

1. Select the filter
2. Drag it up or down to its new position.

Note that you can drag filters inside as well as above or below other filters. When you drag a filter inside another filter, it is called a subfilter. Not all filters allow subfilters. A subfilter is very powerful, because it allows a restricted set of text to be processed.

Deleting Filters from the Filter List

To delete a filter from the filter list:

1. Select the filter
2. Click the 'Remove' button in the lower left hand corner of TextPipe's window or press the 'Delete' key on the keyboard
3. The filter is removed.

You can also shift-click to select a range of filters to delete, or ctrl-click to choose items to delete.

Disabling Filters

To disable a filter:

1. Select the filter
2. Double click it, OR right click it and uncheck the 'Enabled' flag, OR select Toggle filter Enable/Disable from the Edit Menu.

In the next lesson we'll look at

· Adding files.

Exercise 3

1. Load the filter file called 'tutorial\Lesson3.fll'
2. Delete the filter(s) that are shown with delete in their title (ignore the plus signs next to the filter titles)
3. Move the filter(s) to the location shown in their title
4. Disable the filter(s) shown with disable in their title
5. Click the 'Trial Run' button
6. If you've done everything right, the Trial Run output should now display

You've successfully completed

exercise 3!'

Tutorial Index


Lesson 4 - Adding files

 Lesson 4 - Adding files
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

In the last lesson we covered

· How to move filters in the filter list
· How to remove filters from the filter list
· How to disable filters.

In this lesson we'll look at

· Adding files.

Before we begin, open the filter called 'tutorial\lesson4.fll'

Adding files

So far we've just been processing text that's held in TextPipe's Trial Run Area. Now we want to get it to work on some real files, so let's see exactly how we do this:

1. Open Windows Explorer
2. Display the folder 'My Documents\DataMystic\TextPipe'
3. Locate the file called 'tips.txt'
4. Drag and drop the file from Explorer into TextPipe's filter window (with the title 'Untitled..'). If TextPipe's window is not currently visible, drag (and hold the button down) the file to the task bar area where it says 'TextPipe', pause for 2 seconds until TextPipe comes to the foreground, and then drop the file into TextPipe's window
5. TextPipe changes to show the Files to Process tab, with the new file added

It's that easy to add a file or files - just drag and drop them from Explorer. You can also use the [Add Folder] button to add one or more folders, or click the [...] button at the right hand end of the filename field to add one or more files.

To process the file:

1. Click the Go button
2. TextPipe displays the status window
3. TextPipe processes the file, and then plays a sound to let you know it has finished
4. Click on the Results tab
5. Find the line in TextPipe's window (not in the Help) that says 'Output file [My Documents\DataMystic\TextPipe\tips.txt] Size: ...'
6. Double click this line
7. TextPipe opens NotePad to show you the output file.

You can see in the output file that all spaces have been converted to underscore characters '_'. This is what the lesson4.fll filter file is designed to do.

The Subfolders column specifies how many sub folders should be processed.

The Action column specifies what to do with each filename.

In the next lesson we'll look at

· Sub filters (filter cascades)

Exercise 4

1. Modify the lesson4.fll file to replace the underscore characters '_' with spaces, and change tips.txt back to how it was originally.
2. Open the output file and make sure it has spaces in it.

Tutorial Index


Lesson 5 - Sub filters (filter cascades)

 Lesson 5 - Sub filters (filter cascades)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

In the last lesson we covered

· Adding files.

In this lesson we'll look at

· Restrictions and sub filters

What is a restriction?

Put simply, a restriction filter allows only a limited section of text to be processed. It does this using sub filters, which are shown below it and indented to the right in the filter view. A restriction filter to delete 11 lines starting at line 3 might look like this:

A restriction might be lines 100 to 199, columns 30 to 36, lines containing the text 'ERROR', a range of CSV or Tab-delimited fields or a combination of these subsets, such as columns 3 to 17 of lines 20 to 55. Restrictions and sub filters are very handy when you only want to apply a filter to a subset of the text, such as only performing a search and replace within the first 10 lines. Many other variations are possible.

Click here to find out how to add a sub filter

Display of sub filters

TextPipe displays sub filters with a filter hierarchy or filter tree, very similar to the way that Windows Explorer shows drives A: and C: as subsets of My Computer.

When you see a hierarchy of filters in TextPipe you know that a subset of the text is being treated specially. You can imagine the very top part of TextPipe's filter display to be selecting a very large subset - the entire file.

Creating a sub filter

To create a sub filter, you first need to Select the subset of text that you want to treat specially. You do this by choosing one of the filters in the Restrict Menu.

Let's do this now. We will create a filter that will replace the word 'TextPipe' with 'TextPipe Pro' but only in lines 1 to 3:

1. Open the filter called 'tutorial\lesson5.fll'
2. From the Restrict Menu, choose Line Range
3. Customize the Line Range filter so that it will 'Select lines 4..7'
4. From the Replace Menu, add a Find exactly filter
5. Customize the Find and Replace filter so that it will find the text 'TextPipe' and replace it with '*TextPipe Pro*'
6. Drag and drop the Find and Replace filter so that it appears cascaded beneath the Line Range filter. To do this, you need to drag the Find and Replace filter to the right hand side of the tree view before you drop it (or ensure that you are holding down the Shift key). As you do this, the Status Bar will display the text 'Valid Action:Dropping Source filter as sub filter of Target filter'
7. Click the 'Trial Run' button.

After the Trial run completes, you'll notice that only the second and third occurrences of 'TextPipe' have been replaced with 'TextPipe Pro'. This is exactly what we wanted - the Line Range sub filter has only passed lines 4 to 7 onto the Search and Replace sub filter. We can confirm this easily by adding an UPPERCASE filter after the Find and Replace filter.

1. Select the Find and Replace filter
2. From the Conversion Menu, add an UPPERCASE filter
3. The UPPERCASE filter is added immediately after the Find and Replace filter
4. Click the 'Trial Run' button.

With this example we can see that multiple filters can be applied within a sub filter.

Expanding and Collapsing the Filter Tree

Just as you would with Windows Explorer, you can single click the plus sign [+] next to a filter item to expand it out and show its sub filters, and you can single click the minus sign [-] next to a filter item to contract it and hide its sub filters.

Comments

Comment filters can be used to document sub filters inside them. This is handy because you can hide the details of an operation inside a comment filter, and then collapse the comment filter so that the implementation is hidden.

Exercise 5

1. Create a sub filter based on the example above that selects Columns 1 to 35 of lines 1 to 4, and replaces 'TextPipe' with 'TextPipe Pro' within that selection. Only one replacement should be performed.

Tutorial Index


Lesson 6 - Removing text surrounded by other text

 Lesson 6 - Removing text surrounded by other text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Let's say you want to remove some text from a series of web pages or text files. The text is identified by some text that identifies the start and other text that identifies the end, but in the middle the text varies. To match this we need to use a text pattern. TextPipe offers both EasyPatterns and perl patterns (from the perl programming language).

Using EasyPatterns

To add an EasyPattern, select Filters\Replace\Find EasyPattern.

Start text | End text | EasyPattern to search for: | Replace with:
<h1> | </h1> | <h1>[ 1 + chars ]</h1> | <blank>
start | end | start[ 1 + chars ]end | <blank>
<table ...> | </table> | <table[ 1+chars ]>[ 1+ chars ]</table> | <blank>
To capture text so that you can re-arrange or replace bits of it
<h1> | </h1> | <h1>[ capture(1 + chars) ]</h1> | <h2>$1$</h2>
start | end | start[ 1 + chars ]end | newstart$1$newend

If the text to match might be blank, then use '0+ chars' instead of '1+ chars'.

Using Perl patterns

To add a perl pattern filter, select Filters\Replace\Find pattern (perl style). Perl pattern are harder to learn, use and troubleshoot than EasyPatterns.

Start text | End text | Perl pattern to search for: | Replace with:
<h1> | <h1> | <h1>.+</h1> | <blank>
start | end | start.+end | <blank>
<table ...> | </table> | <table.+>.+</table> | <blank>
To capture text so that you can re-arrange or replace bits of it
<h1> | </h1> | <h1>(.+)</h1> | <h2>$1$</h2>
start | end | start(.+)end | newstart$1$newend

If the text to match might be blank, then use '.*' instead of '.+'.

Other Notes

If the variable text is particularly long (e.g. more than 4Kbytes or 4096 bytes), you need to increase the maximum allowed match size by clicking the [...] button beside the pattern type.

Tutorial Index


Lesson 7 - Reformatting text surrounded by other text

 Lesson 7 - Reformatting text surrounded by other text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Let's say you want to reformat some text from a series of web pages or text files. The text is identified by some text that identifies the start and other text that identifies the end, but in the middle the text varies. To match this we need to use a text pattern. TextPipe offers both EasyPatterns and perl patterns (from the perl programming language).

Using EasyPatterns

To add an EasyPattern, select Filters\Replace\Find EasyPattern.

Start text | End text | EasyPattern to search for: | Replace with:
<< | >> | <<[ capture(1 + chars) ]>> | [[$1]]
<start> | <end> | <start>[ capture(1 + chars) ]<end> | <!-- $1 -->
<table ...> | </table> | <start[ 1+chars ]>[ capture(1+ chars) ]<end> | $1

If the text to match might be blank, then use '0+ chars' instead of '1+ chars'.

Using Perl patterns

To add a perl pattern filter, select Filters\Replace\Find pattern (perl style). Perl pattern are harder to learn, use and troubleshoot than EasyPatterns.

Start text | End text | Perl pattern to search for: | Replace with:
<< | >> | <<(.+)>> | [[$1]]
<start> | <end> | <start>(.+)<end> | <!-- $1 -->
<table ...> | </table> | <start.+>(.+)<end> | $1

If the text to match might be blank, then use '.*' instead of '.+'.

Other Notes

If the variable text is particularly long (e.g. more than 4Kbytes or 4096 bytes), you need to increase the maximum allowed match size by clicking the [...] button beside the pattern type.

Tutorial Index


Lesson 8 - Debugging Filters
 Lesson 8 - Debugging Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe offers a number of ways to debug what your filter is doing:

Turn on Prompt on Replace

Turn on the Prompt On Replace option for each search/replace - so you can see what is being found, and what it is being replace with.

When the Prompt window is shown, use the Search Information tab to see which data has gone into which fields.

After debugging, use Tools\Disable Prompting to disable all prompts.

Filter List Right Click/Trial Run (up to and including this filter)

Using the filter menu right click, you can run a filter list just up to a particular point, and see what the text looks like so far. You can also run individual filters and groups of filters.

Using the Debug Filter

The debug filter outputs the text that passes through it, so you can see exactly what the text looks like at each stage.

Tutorial Index


Tutorials and User assistance

 Tutorials and User assistance
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Remember, you can use the Filter Wizard wizard found under Tools Menu\Filter Wizard to easily create new filters.

What do you wish to do?

· TextPipe Tutorial lessons *recommended for beginners
· An Introduction to Pattern Matching
· EasyPatterns
· How to open the example filters
· Step-by-step instructions for using TextPipe
· See common tasks and how to tackle them
· Advanced Topics - automation and troubleshooting


Common tasks

 Common tasks
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Common Tasks

Replacing across a website...

1. Drag and drop your website base/root folder to anywhere in TextPipe's window. It will get added to the Files to Process tab
2. Add a Replace menu\Find exactly filter for each replacement required
3. Click the [Go] button.

You can use a variety of pattern matching filters from the Replace menu. If you have more than one replacement, you can use a Replace menu\Search/Replace list, or import a list.

Grep through website logs

1. Drag and drop your log files or log folder to anywhere in TextPipe's window. They will get added to the Files to Process tab
2. If required, move the files in the file grid to change the order they are joined (or use a Special Menu\Sort filter)
3. Add a Special Menu\Merge filter to join the files together
4. To remove lines you never wish to see, add one or more Remove Menu\Matching lines filter
5. To extract lines matching a pattern, add a Special Menu\Retain Matching Lines filter (note - any non-matching lines are discarded, so you can't really use 2 Retain Matching Lines filters in a row. Instead, use an or '|' operator in the pattern match)
6. Set the Output filter to write all data to a new output file
7. Click the [Go] button.

Fix CSV-delimited or Tab-delimited files

1. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
2. To remove fields, add a Remove menu\Remove delimited fields (CSV, Tab, Pipe etc) filter
3. To copy fields, add a Special Menu\Copy delimited fields (CSV, Tab, Pipe etc) filter
4. To move fields, add a Special Menu\Move delimited fields (CSV, Tab, Pipe etc) filter
5. To replace inside specific fields, add a Restrict Menu\Restrict to delimited fields (CSV, Tab, Pipe etc) filter, then add a Search/Replace sub filter.
6. To convert specific fields to UPPERCASE, add a Convert Menu\Uppercase filter, then add a Search/Replace sub filter
7. ..etc..
8. Click the [Go] button.

Convert from mainframe EBCDIC to ASCII

1. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
2. Add a Convert Menu\EBCDIC -> ASCII filter
3. Set the Input filter to Process Binary files
4. Click the [Go] button.

Join, merge and sort files

1. Drag and drop your files or folders to anywhere in TextPipe's window. They will get added to the Files to Process tab
2. Add a Special Menu\Merge filter to join the files together
3. Add a Special Menu\Sort filter
4. Set the Output filter to write all data to a new output file
5. Click the [Go] button.

Split files

1. Drag and drop your files to anywhere in TextPipe's window. They will get added to the Files to Process tab
2. Add a Special Menu\Split filter to split the files
3. Set the Input filter to Process Binary files
4. Click the [Go] button.

Extract data from a database

1. Add a Special Menu\Database filter. Use the connection string to specify where the database is
2. In the Trial Run Area, enter the text 'select * from tablename', where tablename is the name of your table
3. Set the Input filter to Input from Trial Run Input
4. Set the Output filter to write all data to a new output file
5. Click the [Go] button.

You may wish to customize the output with additional search/replace filters.

Text and data mining

See the sample filter html\data mine html tables.fll for an example of converting a web page to a CSV file for import into Excel. This uses the Extract option of the Search and Replace filter.

Extract email addresses

1. Drag and drop your files to anywhere in TextPipe's window. They will get added to the Files to Process tab
2. Add an Email menu\Extract email addresses filter
3. Add a Special menu\Merge filter
4. Add a Special menu\Sort filter
5. Add a Remove menu\Remove matching lines filter for any addresses you want excluded
6. Set the Output filter to write all data to a new output file
7. Click the [Go] button.

Other tasks

To add a filter, choose one from the menu. It will automatically be added to the filter list. Below are some common filtering operations and where to find the appropriate filter in the menus.

Filter to use:

Find an HTML tag

Use Replace Menu\Find pattern (perl style). Ensure that pattern matching is enabled using the drop down list, and that Minimal matching is specified in the advanced [...] dialog, then enter a search string like this:

<a[^>]*>

This will find all <a> tags. The [^>] means 'find any character except a '>', and the * means 'find 0 or more characters like this one.

Find text within an HTML tag e.g.

<a....height="130"....>

Use Replace Menu\Find inside an HTML tag. Ensure that pattern matching is enabled using the drop down list, and that Minimal matching is specified in the advanced [...] dialog, then enter a search string like this:

<a[^>]*>

This will find all <a...> tags. Now add a sub filter to perform the real search and replace on the text found inside the tags.

Find text between HTML tags e.g.

<a....>......</a>

Use Replace Menu\Find pattern. Ensure that Minimal matching is specified in the advanced [...] dialog, then enter a search string like this:

<a[^>]*>.*</a>

This will find all <a>...</a> tags. Now add a sub filter to perform the real search and replace on the text found between the tags.

Remove HTML

Remove Menu\HTML and XML

Retain the head or tail of a file

This is similar to the Unix command head and tail, use the Remove Menu/Remove start or end of file filter to remove lines or characters from the start or end of the file.

Swap date formats from dd-mm-yyyy to yyyy-mm-dd

Using Replace Menu\Find pattern (perl style), add a search pattern of

(\d+)-(\d+)-(\d+)

and a replacement of

$3-$2-$1

Swap name formats from SURNAME, FIRST to FIRST SURNAME

Using Replace Menu\Find pattern (perl style), add a search pattern of

(\w+), (\w+)

and a replacement of

$2 $1

Convert End of Line characters between Unix/DOS/Mac

Convert Menu\End of Line characters

Search and replace

Replace Menu\Find exactly

Pattern match and replace

Replace Menu\Find pattern (perl style)

Add left margin (indent)

Add Menu\Left margin

Unindent

Use the Remove Menu\Columns filter to remove the first two characters on each line

Delete columns

Remove Menu\Columns

Extract email addresses for mailing lists

Email Menu\Extract emails

Remove blank lines

Remove Menu\Blank lines

Sort

Special Menu\Sort

After the filter is added to the filter list, you can click it to display options in the right hand side of the window. Each filter can be tailored in this manner.


Troubleshooting filters

 Troubleshooting filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Try the following suggestions:

1. Use the partial trial run facility of the Filters to Apply tab to either run individual filters or all filters up to a given point
2. No files processed? Are you processing binary files? If so, check that the Input Filter has the Binary Files option set to 'Process' or 'Confirm before processing'. The status dialog also shows a count of the Binary files skipped in Red so it is easy to tell.
3. Where is your output going? Check that the settings for Test Mode, Change extension to, Output folder, Clipboard output and Single file output on the output filter are set as you expect.
4. Which files and folders are you processing? Check the files tab.
5. Is any input getting through to the output? You can check this by inserting Debug filters at strategic points in your filter to monitor the text at various stages.
6. You can also use the Filter Options to enable logging - this will allow you to see which files are processed, what gets done to them and where they are saved.

If you're running TextPipe from the command line, try the Command Line Helper and the Show Command Line dialogs, and the command line troubleshooting guide.

Stack Overflow Errors

If your patterns are poorly written (or alternatively, you find a bug in the pattern matching library we use), you may run into Stack Overflow errors. These occur when a pattern keeps matching text without any limit, and eventually runs out of memory. They can be avoided by limiting how much text a pattern can match - see Performance. To find the offending filter, use these steps:

1. Load the file into the Trial Run area
2. Right click in the filter list, and choose 'Trial Run up to and including this filter'. Keep altering the filter you click on until the error no longer occurs
3. The filter you just excluded is the problem filter
4. In the problem filter, keep deleting the right-hand end of the pattern until no error occurs
5. The bit you just deleted was the problem. Rewrite it.

You should avoid patterns of the type

([^">])+

where the brackets are unnecessary. Each set of brackets causes a recursive step. This pattern should be written as:

[^">]+

When the match can occur a large number of times, it can often be rewritten as

[^">]{1,10} or ([^">]){1,10}

with no change to the pattern matching process. This avoids too many recursive steps.

If you cannot eliminate the Out of Memory errors or Stack Overflow errors, please send your filter to us.


How can I use TextPipe?

 How can I use TextPipe?
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Its myriad uses include:

· Performing multiple search/replaces on a batch of files - very useful for Web sites.
· Reducing the size of HTML files for faster page loading
· Making HTML files readable without a Web browser
· Converting or reading mainframe, Unix, DOS, Amiga or Macintosh files
· Reformatting text e.g. preparing ASCII files for word processing
· Convert mainframe reports to PC format and copy business user’s PCs overnight in a single step
· Extract data from cumbersome log files
· Organize lists
· Customize text output from other programs
· Reformat program source code
· Automate manual processes
· Interface incompatible software systems
· Convert exported text for use by other software packages
· Format mailing lists for printing
· Search for and/or remove duplicates
· Convert tabular data to delimited and vice versa
· Cross-platform development
· Adding extra word-processing capabilities through converting the clipboard contents
· Making text files readable that contain ANSI escape codes, binary characters or IBM graphics characters
· Adding an email reply prefix to every line
· Converting text from one language to another (e.g. Pascal to c)
· Correcting text files incorrectly FTP'd in binary mode.

See also

What is TextPipe?

Features

New features in this release


Sample filters

 Sample filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe comes with over 100 sample filter list files (.fll files) that solve common text processing problems.

Opening a sample filter

To open a sample filter:

1. Click on Open in the File menu
2. TextPipe displays the filter list files in the TextPipe folder, usually My Documents\DataMystic\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
3. Select a filter list file (.fll file) to open, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html, sysadmin or vbscript and select a filter list file to open
4. Click the Ok button
5. TextPipe will open and display the selected filter list file.

You can now add, delete or modify filters in the displayed filter list.

Inserting a sample filter

Sometimes you need to incorporate the filters that you have saved in a filter list file. To copy them in to the current filter:

1. Position the cursor on top of the filter where you want the inserted filter list to appear. The inserted filter list will appear ABOVE the currently selected filter
2. Click Insert in the File menu
3. TextPipe displays the filter list files in the TextPipe folder, usually My Documents\DataMystic\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
4. Select a filter list file (.fll file) to insert, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to insert
5. Click the Ok button
6. TextPipe will insert the new filter list above the currently selected filter.

Linking to an existing filter (Pro version only)

A linked filter is a copy of a filter list from another file, but with the input and output filters removed so that it sits in the data flow of the filter being linked from. Care must be taken if the Linked Filter contains Scripts that use global variables, to ensure that no accidental overwriting occurs.

Any changes made to an inserted filter list file will not be reflected in the main filter list after is has been inserted. TextPipe provides a linking facility that specifies the file name of an existing filter list file to insert, and the insertion is only performed at the time the filter list is used. This means than any changes that have been made to the linked filter list will be reflected in any filter list that links to it. To link to a filter list:

1. Position the cursor on top of the filter where you want the linked filter list to appear. The linked filter list will appear BELOW the currently selected filter
2. Click Link to filter in the All Available Filters view of the Filters to Apply tab (Pro version only)
3. TextPipe displays the filter list files in the TextPipe folder, usually My Documents\DataMystic\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
4. Select a filter list file (.fll file) to link to, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to link
5. Click the Ok button
6. TextPipe will insert a new link filter BELOW the currently selected filter.


Step by step

 Step by step
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Where is the data that you want to process?

1. in one or more files
2. on the windows clipboard

Note: You can also use the Filter Wizard wizard , found in the Tools menu, to step you through this process.


Step 2a. Process One or More Files

 Step 2a. Process One or More Files
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. Select the Files to process tab
2. You need to add the files to be processed to this window. There are several methods of doing so:
1. Drag and drop files and folders from Windows Explorer
2. Click the Add Folder button to select an entire folder, or to select files from your Network
3. Click the ellipsis button [...] to display a window in which you can select one or more files (press [Ctrl] to select extra files)
4. Click the [Open] icon to open a file list from an existing file
5. Simply type the required filename in the Filename box. The Filename cell supports the standard wildcards that the Windows Find dialog supports.
1. In the TextPipe window, select the Filters to Apply tab
2. Select the File Input or Clipboard input filter near the top of the filter list. The right side of the screen changes to display the options for the input filter
3. Ensure that the Input from file(s) radio button is selected. Input is now set to files to come from the files you have selected.

Note: New rows are automatically added at the base of the grid as entries are added.

Including Subfolders

You can ask TextPipe to scan subfolders by setting the Subfolders column to All, or to ignore Subfolders by setting it to None.

How the Rows are Processed - Include and Exclude

Each row is part of a group of files that are processed together. You can include and exclude files from a group using the Actions Include and Excluce. The exclude action only affects files in the current group, so you can remove files from one file group without affecting similar files in other groups.

For example, lets say you had to process a number of documents in the folders c:\My Documents and n:\clients\documents. Unfortunately, there are some invoices in the c:\My Documents folder that should not be processed, but there are invoices in the n:\clients\documents folder that should be processed. By using a file group, you can tell TextPipe to only remove the invoices in the first folder but not in the second folder. Your file list would look like this:

Note: This screen shot shows an older version of TextPipe where Add=Include and Subtract=Exclude.

Next Step >


Step 2b. Process the Windows Clipboard

 Step 2b. Process the Windows Clipboard
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Input or Clipboard input filter near the top of the filter list. The right side of the screen changes to display the options for the input filter
3. Ensure that the Input from clipboard radio button is selected.

Next Step >


Step 3. Filters to Apply

 Step 3. Filters to Apply
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

Now you must decide what filters to apply to the text. Each filter performs an operation such as a search and replace, adding text to the left margin, converting end of line characters etc, and then passes the result on to the next filter.

The Filters to Apply tab controls which filters get applied and the order that they are applied in. To add a new filter to the list:

1. Place the cursor ABOVE the position where you would like the new filter. The new filter will be added below the cursor
2. Choose a filter to add from the filter menus (Convert, Add, Remove, Email, Replace, Special, Restrict, Maps)
3. The new filter will be added below the cursor
4. Edit the filter's properties on the right hand side of the window.

Below are some common filtering operations and where to find the appropriate filter in the menus.

Common operations | Filter to use:
Convert End of Line characters between Unix/DOS/Mac | Convert Menu\End of Line characters
Search and replace | Replace Menu\Find exactly
Pattern match and replace | Replace Menu\Find pattern
Add left margin (indent) | Add Menu\Left margin
Remove HTML | Remove Menu\HTML and XML
Delete columns | Remove Menu\Columns
Extract email addresses for mailing lists | Email Menu\Extract emails (Pro version only)
Remove blank lines | Remove Menu\Blank lines
Sort | Special Menu\Sort

After the filters have been added to the filter list, you can click on them to display options in the right hand side of the window. Each filter can be tailored in this manner.

Next Step >


Step 4. Where Should the Output Go?

 Step 4. Where Should the Output Go?
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Where should the output be sent?

1. Output to a test run
2. Output back to the original files (overwrite them)
3. Output to a new folder
4. Output to a new file extension
5. Output to the windows clipboard
6. Output to a new file by joining all files together


Step 4a. Overwrite the Original Files

 Step 4a. Overwrite the Original Files
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
3. Ensure that the Output to File(s) radio button is selected
4. Ensure that Test Mode, Change extension to and Output folder are all unchecked.

Next Step >


Step 4b. Output to a New Folder

 Step 4b. Output to a New Folder
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
3. Ensure that the Output to File(s) radio button is selected
4. Ensure that the Output Folder box is checked
5. Select a new folder by clicking the folder icon on the right hand side of the Output Folder box, or by typing a new folder name
6. To maintain the existing folder structure, check the Maintain folder structure check box. To remove the folder structure of the original files, leave the box unchecked.

Each input file is written out to a new folder.

Next Step >


Step 4c. Output to a New File Extension

 Step 4c. Output to a New File Extension
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
3. Ensure that the Output to File(s) radio button is selected
4. Ensure that the Change extension to box is checked.

The original extension of each file is removed and replaced with the given extension. A leading period is assumed if not provided.

Next Step >


Step 4d. Output to a Test Run

 Step 4d. Output to a Test Run
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
3. Ensure that the Output to File(s) radio button is selected
4. Ensure that the Test Mode box is checked.

Each input file is written out to a completely new file, with an extra extension added of .dCv.

Next Step >


Step 4e. Output to the Windows Clipboard

 Step 4e. Output to the Windows Clipboard
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
3. Ensure that the Output to Clipboard (merge) radio button is selected.

If the input consists of multiple files, they will be added together and sent to the clipboard.

Next Step >


Step 4f. Output to a new file by joining all files together

 Step 4f. Output to a new file by joining all files together
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Next Step >

1. In the TextPipe window, select the Filters to apply tab
2. Select the File Output or Clipboard ouput filter at the end of the filter list. The right side of the screen changes to display the options for the ouput filter
3. Ensure that the Single file Output radio button is selected
4. Choose the file where the joined output will be sent.

Next Step >


Step 5. Run the Filter

 Step 5. Run the Filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Congratulations, your filter is now ready to run!

Click the [Go] button to begin processing.

To process text on the Windows clipboard (overriding the Input and Output filter settings) click the [Clipboard] button.

If you wish to test the filter first, select the Trial Run tab, paste or type some text into the Input Text area, then click the [Trial Run] button.


Regular Expression Hints

 Regular Expression Hints
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Introduction

Need we say that regular expressions are a powerful, but sometimes tricky, part of TextPipe. The good news is that you don't need to be a rocket scientist to use regular expressions. The simple ones are just that -- fairly simple -- and they are all you need to understand for many applications. If you're relatively unfamiliar with regular expressions -- regexps for short -- be sure to read the section below Tips on Seat-of-the-Pants Pattern Matching.

If, on the other hand, you are already comfortable using regular expressions, you won't find much of value in the Tips section, but may find subsequent tables and examples useful. Our focus is on items, like quantifiers, that you keep looking up in your reference materials because you don't use them often enough to remember the fine points.

Tips on Seat-of-the-Pants Pattern Matching

These are tips, not a tutorial. We assume you already know how to code simple substitutions with regular expressions.

The irony of pattern matching with regular expressions is that although it's a highly deterministic operation, this doesn't help you much unless you are a rocket scientist. The pattern matching engine uses a complex set of rules. But, the rules are hard to understand and difficult to apply to practical problems of some complexity -- even for full-time Web page developers, much less part-timers. Fortunately, you can mostly forget about the formal matching rules and take a much more practical or seat-of-the pants approach.

With this as background, here are our tips:

1. Never overwrite your input file. There's no Undo function as there is with your favorite word processor. It's easy, oh so easy, to make coding errors in substitution statements, even when you have experience coding them.
2. If you're using the don't-care character sequence (".*" or ".+") in your pattern, watch out for greedy matching -- which typically matches the right-most pattern rather than the left-most. See the section Quantifiers for examples.

Metacharacters

The following characters -- so-called metacharacters -- have special meaning in regular expressions. To match them, precede them with a backslash.

Metacharacter | Purpose
\ | first character of assertions, such as \w or \d
| | OR; allows matching options
. | match any character
( ... ) | grouping operator; builds $1, $2, etc.
[ ... ] | match any character within brackets
+ | quantifier: match one or more times
? | quantifier: at most one match
* | quantifier: match zero or more times

Assertions identify special conditions for a pattern match. The following table covers commonly used assertions.

Common Assertions
Assertion | Function
\d | digit [0-9]
\d+ | number
\D | nondigit
\s | white-space
\S | non white-space
\w | alphanumeric character [a-zA-Z_0-9]
\w+ | alphanumeric word
\W | nonalphanumeric character

Quantifiers

This section complements discussions of quantifiers that are long on words but short on examples. We examine a host of variations on a basic example, using the various quantifiers in the following scenario:

Search for | (pe)
Replace with | --
Target string | Pepe pens Perl pearls.

We use parentheses in all examples for consistency. They only matter here when followed by a quantifier. Thus (pe)+ looks for successive occurrences of pe, whereas pe+ looks for successive occurrences of e.

Substitutions shown in red merit close attention.

Search string | Case sensitive? | Result | Notes
No quantifier (i.e., exactly one match) |  | Pepe pens Perl pearls. |
(pe) | Y | Pe-- --ns Perl --arls. |
(pe) | N | ---- --ns --rl --arls. |
Plus quantifier (one or more matches) |  | Pepe pens Perl pearls. |
(pe)+ | Y | Pe-- --ns Perl --arls. |
(pe)+ | N | -- --ns --rl --arls. |
Question mark quantifier (at most one match) |  | Pepe pens Perl pearls. |
(pe)? | Y | --P--e---- ----n--s-- --P- etc. |
Asterisk quantifier (zero or more matches) |  | Pepe pens Perl pearls. |
(pe)* | Y | --P--e---- ----n--s-- --P- etc. |
Leading don't-care characters |  | Pepe pens Perl pearls. |
.*(pe) | Y | --arls. | Greedy match
.*(Pe) | Y | --rl pearls. | Greedy match
.+(Pe) | Y | --rl pearls. |

See also

Regular expressions


Features

 Features
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe can perform automated or manual line-feed conversion to and from DOS, Unix, Amiga and Macintosh input formats, and convert Mainframe fixed-length records (with a user-specified length) to separate lines.

To make files smaller and/or more readable, TextPipe offers filters to:

· remove binary characters
· remove ANSI escape sequences
· remove blanks from the start and end of lines
· remove blank lines
· compress multiple whitespace to a single space
· remove HTML and SGML codes
· remove or resolve backspaces
· change IBM drawing characters to + and | and -

To manipulate the text in files, TextPipe offers filters to:

· perform an arbitrary mapping. The user can specify a string to be output for each of the 256 possible character codes. Strings can contain embedded control characters
· search and replace: with case sensitivity option. It gives the length of search and replace strings -- especially useful for text replacement in binary files
· add a prefix to every line (eg to add reply quotes to email)
· change character case: UPPERCASE, lowercase, tOGGLE cASE, Title Case, Sentence case, rANdOm CAse
· change character representation between ASCII and EBCDIC. Useful for mainframe interfacing
· change between Windows OEM and ANSI. Useful with Windows and non-English languages

TextPipe offers the following file processing options:

· can skip binary files. The user can also be prompted whether to process a binary file or not
· can prompt before processing each file
· can retain the original file's time and date
· can recurse into subfolders
· can process both text and binary files of any size
· test mode where output goes to a new file rather than overwriting the original file

Special features include:

· Filter lists can be saved and opened
· Files can be quickly viewed with their associated application
· Many methods of adding files to the list - selected via wildcards on the command line, specified in the application, drag and drop from explorer or via a Browse dialog
· Accept a list filename on the command line to include large lists of files that couldn’t otherwise be specified. Can be used in conjunction with the Open List From File and Save List To File icons.
· Accepts wildcards on the command line and in list files. The wildcards are expanded during processing.
· No limit on input file size

TextPipe runs under Windows 95/98/ME/NT/2000/XP.

See also

What is TextPipe?

New features in this release

How can I use TextPipe?


Consulting Services

 Consulting Services
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Professional Services

DataMystic offers a range of Professional Services to our clients. Our services include:

· Adding new features to our existing applications
· Creating new applications
· Creating customized, one-off solutions based on our existing applications
· Consulting services to tailor the processing of TextPipe, ResumePipe, DataPipe and our other applications
· Fixing and converting data files
· Data mining web-based data
· Fixing junk data in databases
· Processing emails.

Typical projects that we have been engaged on include

· Optimizing TextPipe filters for the greatest speed
· Extracting data from websites to a database import file, or to an Access database
· Reformatting corporate data to allow import or further manipulation
· Removal of binary or garbage data from files
· Writing VBScript/JScript code to perform custom file manipulations.

For more information, please click here.


New features in this release
 New features in this release
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

What's New in TextPipe v9.7.3 – August 22, 2014
===============================================

*
Removed warning about active COM objects.

What's New in TextPipe v9.7.2 – August 15, 2014
===============================================

*
Fixed bug in Remove lines matching perl pattern.

What's New in TextPipe v9.7.1 – August 14, 2014
===============================================

*
Difference tool integration - new function to open a file in text difference
tool – in Trial Run, from the Files Tab, and from the Status Results Window.
Integrate with tools such as WinDiff and BeyondCompare.
*
Sort function now much faster.
*
Removed locking of TextPipe into the COM Running Object table, so COM errors
do not occur on completion of scripts.
*
Upgraded PDF component.
*
Display text now outputs 'simultaneous search' and 'Process longest strings
first' options.
*
Fixed bug in Convert Excel to Text filter – was ignoring non-hidden sheets.

What's New in TextPipe v9.7 – May 29, 2014
==========================================

*
Mainframe copybook filter now allows pass-thru EBCDIC data ie BINARY data
e.g. using PIC X(48) USAGE IS BINARY.
Mainframe copybook filter now does not insert a null (\x00) character after
fields in Fixed Width mode.
Invalid copybooks now send errors to log when running from command line.
*
New EasyPatterns:
    [Drive]    e.g. C:   E: -> returns drive letter in @drive@
    [Folder]   e.g. \My Documents\  -> returns 'My Documents' in @folder@
    [Path]     e.g. d:\My Documents\  -> returns 'd:\My Documents' in
                    @drive@ and @path@
    [UNCPath]  e.g. \\server\share\path\filename -> returns @server@,
                    @share@ and @path
    [Filename] e.g. testdocument.doc -> returns @filename@
*
Unicode conversion filter did not retain the Error Character if the Input or
Output encodings were not changed.
*
Hidden worksheets are now ignored by the Excel to Text filter.
*
Split filter now cleans up its temporary files.

What's New in TextPipe v9.6 – January 1, 2014
=============================================

*
Trial run area now shows output progressively – giving important feedback
earlier in the filter development lifecycle.
*
Now shows '(more pending)' next to the file count if more files remain to be
counted, so users aren't misled that only 1000 files will be processed.
*
Added new filter list menu items for Remove exactly, Remove perl pattern, and
Remove EasyPattern
*
File grid now allows mass-changing of Action or Subfolder columns using
right-click menu after selecting rows in the row number column.
*
File grid ensures that the top blank row is used with drag/drop.
*
Pressing [Tab] in a memo control now inserts a tab – it no longer changes
focus to the next field.
*
Updated Perl regexp third-party package.
*
Updated PDF third-party package.
*
Re-ordered scratch pad and gave tabs consecutive numbers.
*
'Delete' action of search/replace grid now much faster for entire-grid
clearing.
*
TextPipe now warns if a Convert Word/Excel to Text filter is used at the same
time as the Input Filter/Process Inside Compressed files option – these are
not compatible.
*
Fixed missing comma in VBScript/Jscript code generation for search/replace
lists.
*
Installer - fixed x32 shell extensions missing under Windows x32.
*
Replace List path is now relative to the filter path.
*
Using File Menu\Open to open non-filter files, now adds them to the current
Files To Process list.

What's New in TextPipe v9.5.3 – August 5, 2013
==============================================

*
Fixed error with Restrict to Blocks of Length X and several other filters
causing out of memory errors, as they were confused for 'repeat filter X
times' filters.

What's New in TextPipe v9.5.2 – July 25, 2013
=============================================

*
Fixed 'Only log when a replace occurs' not working for consecutive search
replace filters in the Search/Replace list filter.
*
Fixed UTF-8 mode perl pattern replace strings from being converted to utf-8.

What's New in TextPipe v9.5.1 – July 15, 2013
=============================================

*
Added Help entry for Repeat Subfilters X Times.
*
Adding filters with no valid filter window now creates a new filter window.
*
When Utf8 Support is checked for replace filters, Unicode characters in the
search for and replace with fields are now correctly converted to UTF-8 prior
to being processed by the PCRE engine.
*
Clicking [Delete] on empty areas of the Files grid no longer causes errors.

What's New in TextPipe v9.5 – July 3, 2013
==========================================

*
Added log entries for size/date of ignored files.
*
Search/replace lists can now generate log entries – useful for debugging.
Also, logs can be output only for where replacements occurred, which keeps
the log size manageable when importing large search/replace lists (default On
when loading existing filters as previously no logging was done).
*
EasyPatterns v2.8 – new keywords EBCDICletter, EBCDICupper, EBCDIClower,
DayNameShort, DayNameLong, MonthNameShort, MonthNameLong, YearShort,
YearLong, Float, Integer. WordBreak is a new synonym for WordBoundary. The
behaviour of number, numbers and numeric has been changed so that instead of
matching a single digit (which is already handled by the digit keyword), they
now match a number with an optional sign and optional decimal point.
*
Mainframe copybook filter now copes with VALUES of ZERO/ZEROS/ZEROES,
SPACE/SPACES and multiple comma-separated values in addition to the string
and integer values previously supported. Now supports VALUE clause on numeric
fields.
*
New right-click menu items in all filters view, to Link To or Insert filter
files.
*
Filter comments can now be included in the output log, from the filter
options settings at the top of the tree. This greatly assists in debugging
complex filters.
*
New filter to Repeat Subfilters X Times; useful when you are not sure how
many times a search/replace pattern needs to be applied.
*
Named subexpressions created by a Split on Pattern filter are now saved as
global variables for use in other filters. Useful for pushing split pattern
text into filenames.
*
If Prompt On Replace is disabled during a filter list, the filter is now
marked as modified when the Prompt form is closed.
*
Search/replace lists now discard blank search terms and terms where the
replacement is identical to the search.
*
Log filename now has environment variables resolved before display.
*
Partial trial runs now reset the status window properly.
*
Better handling of split filenames with format string errors in them.
*
Filter components relative to the filter path now work in Unicode folder
names.
*
Restrict to Inside filenames is now available in the Lite version to assist
batch file re-naming.
*
Upgraded PDF component from 8.15 to 9.13 – now handles a greater range of PDF
document types.
*
Updated to Zip Master 1.9.1 and PCRE v7.5.
*
Fixed handling of EasyPattern Ascii() keyword.
*
Fixed bug with Right-click\Delete on files in the file list.
*
Fixed bug in search/replace list, in perl pattern match mode. If the search
term contained brackets (), then replacement phrases on rows following it
were mismatched.
*
Split filter now copes with absolute pathnames in the split filename. Split
filter now outputs log entries when it starts a new file, and outputs
separate logging details for each split file.
*
Number of files processed now does not include files inside zip containers.
The current file being processed now shows the zip container and the internal
filename.
*
Fixed export of Search/Replace lists filters to VBScript.
*
Fixed incorrect path for shell extensions on install.
*
Fixed bug with handling of moving a delimited field, where the line did not
end with the expected delimiter.

What's New in TextPipe v9.4.1a – April 18, 2013
===============================================

*
Bug fix for Retain modified dates for the compressed file.
*
Matrix search now uses slightly faster initialization method.

What's New in TextPipe v9.4.1 – April 9, 2013
=============================================

*
Fixed installer issue on x32 systems.
*
Retain modified dates now works for compressed files (for the compressed file
itself).

What's New in TextPipe v9.4 – April 3, 2013
===========================================

*
Matrix (parallel) search/replace for Exact replacements:
- now supports First Only option – any term that has already been replaced
will not be replaced again.
- now supports Case Sensitive replace- where the case of the replacement is
based on the case of the found text.
- now supports prompt on replace.
- now much faster - 600,000 replacements against 8MB compressed .xlsx file in
31 seconds.
- fixed bug where changes were not counted and hence the output file was
never updated unless changes had been made by other filters.
- fixed bug where text that did not match was output in lowercase when Match
Case was unchecked.
- fixed bug where text at the end of the file or restriction was not
replaced.
*
Added .XPS as a compressed document type.
*
Word to Text Filter and PDF to Text Filter now set output filename to .txt,
Excel to CSV filter now sets output filename to .csv.
*
New warnings to ensure Process Binary files is checked when Process Inside
Compressed Files is checked.
*
TextPipe now switches immediately to the Status tab when a trial run is
started so that progress can be reviewed, and then switches back to the Trial
Run tab on completion.
*
Exact search replace now does not even allow special characters.
*
New status messages from search/replace list filters to give progress updates
when loading large files.
*
New filter to restrict processing to specified files within a zip or other
compressed file.
*
Compressed file handler now removes any left over files at the end of the
filter job.
*
Compressed file handler now shows status information as it extracts and re-
compresses files.
*
New status logs for filters that perform end-of-line conversions.
*
New validation check for 'Send variable N to subfilter' to ensure there are
at least N captured expressions.
*
Now handles any PKZIP compressed file types regardless of whether the
extension is known or not.
*
Fixed bug with handling compressed files where modified files were not
updated back into the original compressed file.
*
Fixed bug where AV occurred when adding a new file to a file list that had
been cleared using [Clear All].
*
Fixed export to clipboard of search/replace lists – the second and subsequent
lines were not indented correctly.
*
Removed 'Warning: Find and Replace string lengths don't match' message when
the Action is not set to 'Replace'.

What's New in TextPipe v9.3.1 – January 8, 2013
===============================================

*
Fixed problem with endless loop caused when Up or Down arrow pressed in Files
Grid.

What's New in TextPipe v9.3 – December 18, 2012
===============================================

*
File grid now supports Unicode characters.
*
Support for Unicode pattern matches in the files grid (perl regex mode).
*
Split filter now allows Unicode filenames, and Unicode file break patterns.
*
Grep filters now allow Unicode patterns (when UTF-8 support mode is enabled).

What's New in TextPipe v9.2.2 – September 6, 2012
=================================================

*
Fixed installer error with registry entries required by x64 Shell Extension

What's New in TextPipe v9.2.1 – September 5, 2012
=================================================

*
Fixed crash with sorting more than one file.
*
Fixed crash experienced by some users caused by the x64 shell extension when
the desktop was right-clicked.

What's New in TextPipe v9.2 – August 12, 2012
=============================================

*
Massive performance improvement for large added strings, such as headers and
footers.
*
Mainframe copybook filter now handles the option 'Process numeric fields
only' properly – previously it output blanks for alpha fields in this mode.
*
New popup menu for the status log – options to copy the log entry, copy all
log entries, open the referenced file or its containing folder.
*
New filter Convert HTML/XML Entities to Text, as it handles both numeric and
253 named entities.
*
New 'Copy to Clipboard' button for Search/Replace list filter.
*
Right-click menus now display available environment variables and their
current values, for easy insertion.
*
Updated internal PCRE (Pattern Matching) engine to v8.30 and support for
Unicode 6.1.0.
*
The perl regex position '^' for the start of the file now works with Unix
files.
*
TextPipe Engine now returns an error if the output file is open in another
application – in the same way the GUI does.
*
Improved COM object model to have less embedded references to objects, making
it easier to free the objects. Also, calling the output* and input* functions
when calling the TextPipe.FilterWindow object no longer result in errors.
*
Uninstall from now on will not remove settings.
*
'updateDate' in registry can now be set to zero to disable update checking.
*
TextPipe Engine now ignores Ctrl+C and Ctrl+Break.
*
MIME Base 64 Decoder now excludes the trailing garbage character.
*
Output Filter now allows macros and ENV vars in the single output filename.
*
Fixed threading errors when cancelling running jobs.
*
Debug window and embedded help windows now record their size correctly.
*
Stack overflow, as in the case of pattern matches with poor constraints, are
now trapped as Exit Code 254.
*
Fixed environment variable substitution error in filenames.

What's New in TextPipe v9.1 – March 15, 2012
============================================

*
Added new macro @inputPath to contain the path to the input file, with a
trailing slash. This lets you easily create output folders at the input path
location.
*
Added new macro @inputExtension to contain the extension of the current input
file.
*
Output Filter's Single Output Filename, new File Extension and New Path can
now all contain macros.
*
Added support for Open Office Writer (.ODT), Calc (.ODS) and Impress (.ODP)
file types as new compressed file types.
*
HTML Restriction filter now supports entity names with embedded colons (:)
and hyphens (-) which is essential for working with Open Office files and
other XML types.
*
EasyPattern keyword [backwardSingleQuote] renamed to [graveAccent] as the
original name was incorrect.
*
The right-click context menu on the Filter Tree now allows groups of selected
filters for 'Make Into a Subfilter' and 'Promote'.
*
Fixed operation of log 'overwrite' option – the log cache would not clear
previous logs created since TextPipe was started. Logs were also not being
closed at the completion of a filter job, hence they could not be deleted.
*
Macros and environment variables are now allowed to contain numbers, provided
that the first character is not a number.
*
The right-click context menu of the replace string now inserts captured
variable names instead of $1, $2 etc when you ask it to insert a CSV, Tab or
XML string. This makes it far easier to identify which fields belong where,
as well as making the replacement string less sensitive to changes to the
search pattern.
*
New optimization for named captured variables – TextPipe replaces these with
the captured variable number behind the scenes – so you get the ease of use
benefits of named variables, with no speed decrease.
*
EasyPatterns now allow a literal (ie a string of characters) inside a 'not'
expression, e.g. not 'ABC', or not EBCDIC('HDR').
*
Mainframe copybook filter can now remove FILLER fields inside OCCURS blocks.
*
Supported but not installed code pages now show 'NOT INSTALLED' beside them.
*
Fixed AV's when saving or loading a file when running under Windows 7.
*
Sort filter set to maximum allowed memory of 2GB to avoid integer overflows.

What's New in TextPipe v9.0 – January 30, 2012
==============================================

*
New filter to perform a calculation on restricted text e.g. add 1.5 to a
field.
*
Comment filter now allows Unicode text.
*
Windows Home Server no longer considered a 'Server' platform.
*
Log files now support Unicode.
*
Fixed action of 'Skip/Replace in Rest of File' when used inside a restriction
– previously it carried forward for the rest of the job, instead of the rest
of the current file.
*
Remove multiple whitespace filter now generates detailed log information.
*
Trial Runs now generate the same logging data as real runs.
*
Fixed missing filters from Lite Edition.
*
TextPipe only registers its COM object during install and for the /REGSERVER
command line option – preventing errors on less-than-admin use.
*
Restored action of F9 button to start running when the All Filters view is
selected.
*
Opening and Updating zip file log messages are now clickable.
*
Updated third-party zip DLL to improve Windows x64 support.
*
Fixed handling of double backslashes in the replacement term for
search/replace lists when consecutive search was selected.
*
Unicode strings are now converted to UTF8 when the perl pattern filter has
UTF8 mode selected.
*
Restrict to matching/non-matching filenames now works with files of unlimited
size.

What's New in TextPipe v8.9.9 – November 10, 2011
=================================================

*
Fixed Sort filter allowing duplicates.
*
Entering \x{1234} when UTF-8 mode is enabled for a perl pattern match now
does not show a validation message.
*
Updated PCRE engine.
*
Fixed scrolling of found text into view in Prompt On Replace dialog.
*
New [Add Files] button on Files to Process tab.
*
Combo boxes now work with right-click context menus.
*
Search/replace list import now supports Unicode.

What's New in TextPipe v8.9.8.1 – October 24, 2011
==================================================

*
Updated PDF libraries for extraction from more recent file formats.
*
Added large [Add Files] button to Files to Process tab.

What's New in TextPipe v8.9.8 – October 24, 2011
================================================

*
Capture Text, Break on Value Change window now shows length of strings and
current cursor position.
*
Updated internal PCRE (Pattern Matching ) engine to v8.13 and support for
Unicode 6.0.0.
*
Updated Unicode internal libraries to support Unicode 4.1 for Normalization
etc.
*
COM callees are now notified of Stack violations (e.g. during pattern
execution) or other critical errors via the existing 'FilterWindow.errorText'
variable.
*
Only modified files are added back into Zip files such as .zip, .docx, .xlsx
and .pptx.

What's New in TextPipe v8.9.7 – October 13, 2011
================================================

*
Updated x64 extension to only show current file name, not full path.
*
Better checking for changed files with Extract matching/non-matching lines
and Remove matching/non-matching lines.
*
All Available Filters tree now show properly when the text 'Search...' is
translated.
*
Fixed errors with TP Engine not unloading cleanly, and not returning data.

What's New in TextPipe v8.9.6 – September 15, 2011
==================================================

*
Windows Explorer x64 extension working!

What's New in TextPipe v8.9.5 – September 2, 2011
=================================================

*
Fixed dimming of unavailable filters in Lite and Standard versions.

What's New in TextPipe v8.9.4 – August 18, 2011
===============================================

*
Fixed Browse buttons not resizing correctly.
*
Converted file list field to Unicode.
*
New 'Close' buttons in basic forms (editor, copybook tree viewer).
*
Now handles 'Convert Word/Excel to text' inside subfilters, prior to the job
starting (e.g. for processing exclusion lists etc).
*
Fixed Mainframe copybook filter handling of OCCURS .. DEPENDING ON .. when
the Trim Fields option was checked.
*
Fixed display of Unicode text in various multi-line edit fields.

What's New in TextPipe v8.9.3 – July 26, 2011
=============================================

*
Fixed crash on filter run when working with files.
*
New 'Open Containing Folder' for filter lists.
*
Fixed bug when double-clicking on filenames.
*
Filter search now works for filters like 'Sort' – where previously the
invisible short cut character & prevented this from working.
*
Search/replace lists now returned to naïve UTF-8 handling.
*
Binary test now works for Unicode filenames.

What's New in TextPipe v8.9.2 – July 19, 2011
=============================================

*
Fixed bug with the way that PlaySound was called at the end of the filter job
which caused crashes.

What's New in TextPipe v8.9.1 – July 8, 2011
============================================

*
Removed windows explorer x64 extension as buggy.
*
Search/replace lists now check for Unicode external files.

What's New in TextPipe v8.9 – July 8, 2011
==========================================

*
New search function to quickly find matching filters or external files.
*
Simultaneous search now works with patterns – not just Exact match.
*
Filter move Up/Down buttons in short cut menu.
*
Enable/Disable prompting in short cut menu to easily debug large groups of
filters.
*
Secondary output filters are no longer removed when pasting.
*
Log files are now opened in their associated app rather than in Notepad when
double-clicked or when [Show] button clicked.
*
Better Unicode filename support when filenames are launched by double-
clicking on text fields.
*
Updated internal pattern matching libraries.
*
TextPipe Engine now logs just as the GUI can.
*
Unicode filenames now supported by Merge Filename; Output Extension, New Base
Folder and Single Output Filename; Split Filename; Log filename; Map comment;
String filters' filenames, Search/Replace List filename.
*
Unicode filename support in status grid, file grid, processing etc.

What's New in TextPipe v8.8.2 – April 13, 2011
==============================================

*
Dimmed filters now cannot be double-clicked to add them.
*
While loading a search/replace list, TP now responds to events.
*
Link Filter now in every edition.
*
List filename now gets loaded properly.

What's New in TextPipe v8.8.1 – April 11, 2011
==============================================

*
Added .ODT as a compressed file type so that OpenDocument documents can be
handled just like Office .DOCX format compressed XML files.
*
New Files to Process field to load files from a list file (note – this
ability was already available from the command line with /L= and from COM via
addFileList).
*
The Replace List filter now has a button to paste a list from the clipboard.
*
Added new exit code of 255 for abnormal termination due to textpipe.exe being
killed or an unexpected error.
*
Deferred hooking into Windows HTML help system for faster startup.
*
Migration of filters from Pro to Standard, and Casing filters from Standard
to Lite.

What's New in TextPipe v8.8 – March 24, 2011
============================================

*
New file date and file size filters to help select files to be processed.
*
New context menu items for filter list, to duplicate, demote or promote
filters.
*
Renamed variables in Extract-generated .vbs and .js scripts for consistency
and readability.
*
Lite and Standard version now show dimmed menu items for unavailable options.
*
Upgraded PCRE pattern engine.
*
Improved speed of double-quoting and trimming mechanism in Mainframe Copybook
filter.
*
New EasyPatterns [DayNameLocal] and [MonthNameLocal].
*
Default for loading older Mainframe Copybook filters without Force Quotes
options is now set to unchecked.
*
Fixed EasyPattern keyword 'atomic'.
*
Fixed coInitialize/coUninitialize COM errors with repeated running of the
scripting filter.

What's New in TextPipe v8.7 – February 14, 2011
===============================================

*
Comment filters and other filters with multi-line values now display just the
first non-blank line in My Filter List.
*
Various GUI fixes.
*
New macro @outputFilename to output the name of the output file without the
path.
*
Correct detection of Windows XP x64 as a non-server O/S.
*
Updated code to detect My Documents and AppsData folder under Windows 7.
*
Convert Excel to Text filter now outputs a form feed (ASC 12) character
between worksheets.
*
Filter lists are now freed on completion. E.g. Filters that load large
suppression lists (e.g. 300MB Remove matching lines filters) can now be run
repeatedly without restarts.
*
Better error trapping for Out of Memory errors and other exceptions that
occur during filter execution.
*
Many memory leaks fixed, making working with large files multiple times far
easier.

What's New in TextPipe v8.6.8 – November 16, 2010
=================================================

*
Speed improvements for Remove Lines From List, and Retain Lines From List,
when using large external compare files (10MB+).

What's New in TextPipe v8.6.7 – October 27, 2010
================================================

*
The Output Filter setting of 'Backup Mode' now does indeed rename the
original input file to '.bak' rather than operating on the output filename.

What's New in TextPipe v8.6.6 – October 19, 2010
================================================

*
Setting the global variable @CLIPBOARD using a Capture Filter or a named
variable in a search pattern now sets the Windows clipboard as well.
*
Missing Recently Used files are now removed from the list.
*
New 'atomic' keyword allows EasyPatterns to match either all or none of a
group e.g.
  [ atomic(0+ spaces) ]
matches all the spaces that are present, or none of them – and does not
backtrack and keep retrying with one less space each time. This makes
matching far faster.
*
Mainframe Copybook Filter in the Server Edition no longer automatically adds
a Split CPU filter.

What's New in TextPipe v8.6.5 – September 8, 2010
=================================================

*
Shell extension now supports Unicode filenames – fixed error with only the
first letter of menu options being shown.

What's New in TextPipe v8.6.4 – August 25, 2010
===============================================

*
Mainframe copybook filter now allows trailing minus sign to identify signed
fields e.g. PIC 9(11)V99-.
*
Fixed problem where status messages were occasionally lost or doubled up.
*
Improved Unicode file handling.

What's New in TextPipe v8.6.3 – August 13, 2010
===============================================

*
Fixed bug with output files causing file rename errors.

What's New in TextPipe v8.6.2 – August 11, 2010
===============================================

*
Sound on completion when run from command line now uses settings from
Preferences (rather than being forced on). Can be overridden using
/PLAYSOUND.
*
Updated Explorer Context Menu to support Unicode.
*
Now uses updated zip library.
*
Partial support for Unicode filenames.
*
Fixed bug with Ignore Case always enabled, for Count Duplicate Lines and
Remove Duplicate Lines filters.
*
Fixed bug where @ and $ signs in the replace string were handled incorrectly-
when used in the Search/Replace list filter in sequential mode.

What's New in TextPipe v8.6.1b – July 2, 2010
=============================================

*
Change to update mechanism to prevent spurious update notifications.

What's New in TextPipe v8.6.1 – July 1, 2010
============================================

*
Log file is now closed and re-opened every 30 seconds so that the file size
increases over time. If you open the file it will still show the currently
written data.
*
Fixed input file gathering – new files are now available for processing
immediately rather than grouped by folder. Previously, folders with thousands
of files would cause an apparent stop in processing.
*
Uninstaller now does not require explorer.exe (Explorers and Task bar) to be
closed; reboot or logoff required as before.
*
Cancel button now does not pause thread until [Ok] clicked, due to conflicts
with suspending the thread.

What's New in TextPipe v8.6 – June 16, 2010
===========================================

*
File list is generated in parallel with processing – so work commences
immediately, even with millions of files to process.
*
Status window now shows expected completion time based on total data volume
being processed.
*
Mainframe Copybook Filter – adding quotes to alpha fields is now an option
rather than automatic. Quotes and field delimiters in the record header and
numeric fields are now escaped appropriately.
*
'Delete Input Files On Completion' Input Filter warning message now disabled
in command line and Engine mode.
*
Scripting filter now available in the Standard version.
*
Performance improvement for Switch CPU filter.
*
Restrict to matching lines filter now uses its subfilters to determine if a
change has occurred to the input file.
*
Fixed bug with Split CPU filter when used with multiple output files.

What's New in TextPipe v8.5.3 – May 19, 2010
============================================

*
Fixed bug in MultiCPU version with Split CPU filter stopping output files
being written.
*
Merge filename is now added to the COM output files list.
*
Fixed TextPipe Engine bug with Edit Distance search/replace and Convert to
Word List not working on first filter execution.
*
Fixed Insert Filter option changing filename and inserting extra Output
filter.

What's New in TextPipe v8.5.2 – May 11, 2010
============================================

*
If the Output file cannot be written to (e.g. the network resource is no
longer available), then the Input filter option to Delete Input Files is
ignored. The Open Output Files On Completion option is also ignored in this
case.
*
When run from the command line, errors (such as no matching files to process)
now do not require manual intervention. They are available in the log if it
is turned on.
*
If TextPipe cannot create a temporary file on the target drive (e.g. if it is
a network drive), then it falls back to creating it on the local machine in
%TEMP% and copies it across at the end of the job.

What's New in TextPipe v8.5.1 – April 15, 2010
==============================================

*
Fixed bug with HTML filter when 'Include text' is unchecked, the restriction
is designed to match text between open and close tags, and the open tag has
no text after the tag name e.g. ...

What's New in TextPipe v8.5 – April 7, 2010
===========================================

*
Split filename can now include macros using @varname syntax.
*
Database filter now copes with multiple record sets from a single query.
*
Changed installer to not require admin, for Vista and Windows 7.
*
TextPipe now does not require admin rights to run – which in Windows
Explorer:
- restores ability to double-clicking a filter to open it
- restores the TextPipe context menu
- restores drag and drop.
*
Clipboard input and output can now both be in Unicode format.
*
New COM method terminate() to allow filters to be stopped when called from
execute() or executeClipboard().
*
TextPipe Engine now supports the Split CPU filter.
*
Fixed bug that occurred with T-Filters and Uppercase/Lowercase filters. The
Upper/Lower filters overwrote memory so that modified text could occur in the
alternate stream of the T-Filter.
*
Fixed filter 'Restrict to filenames matching pattern'.

What's New in TextPipe v8.4.9.1 – February 24, 2010
===================================================

*
Mainframe copybook filter – now displays a red bar at column 72 – any data
beyond here gets truncated in line with mainframe commenting standards.
*
COM function addSimpleFilter() now exports the isUnicode option.
*
Much more complex pattern matches now allowed - increased limit of capturing
patterns from 99 to 65535.
*
Removed environment variable capability from the split filename because of
conflicts with %f, %e and %d macros for the file name, extension and number.

What's New in TextPipe v8.4.9 – February 18, 2010
=================================================

*
Mainframe copybook filter – alpha fields now have quotes added around them,
and existing quotes are doubled up to escape them.
*
Environment variables now allowed in the split filename.

What's New in TextPipe v8.4.8 – December 3, 2009
================================================

*
Added [alphanumeric] and [numeric] to EasyPatterns.
*
Fixed validation when a Restrict to Filename NOT matching filter was missing
subfilters.
*
Fixed bug in invocation from shell extension.

What's New in TextPipe v8.4.7 - November 27, 2009
=================================================

*
Modified TextPipe and TextPipe Engine so that error messages are now
available through the ErrorText variable when called via COM.
*
Output file size was missing from the log.

What's New in TextPipe v8.4.6 - November 24, 2009
=================================================

*
Fixed Unicode Search/Replace error that caused file corruption beyond
64Kbytes.

What's New in TextPipe v8.4.5 - November 20, 2009
=================================================

*
Fixed error with clearing of output strings.

What's New in TextPipe v8.4.4 - November 18, 2009
=================================================

*
Script errors that occur at Run Time are now logged as warnings in the log
rather than stopping the filtering.
*
Fixed error with parsing of output strings.

What's New in TextPipe v8.4.3 - November 5, 2009
================================================

*
Fixed shell extension not starting filters correctly.

What's New in TextPipe v8.4.2 - October 29, 2009
================================================

*
New 'Prepare Query' option for database filter to make SELECTs more
compatible.
*
Fixed elapsed time display when duration was less than 1 second.
*
New Edit Menu\Preference to control if Word Wrap is on by default in edit
controls.
*
New command line option /fromshellext to indicate if a filter is being run
from the Windows Explorer shell extension. Existing files in a filter
triggered from Windows Explorer are now cleared and a warning given.
*
Added new COM function addMainframeCopybookFilter() to recreate Mainframe
Copybook Filters.
*
New installer should allow overwrite of existing installs and/or uninstall
without reboot.

What's New in TextPipe v8.4.1 - October 13, 2009
================================================

*
Files to Process, and Search and Replace List filenames can now be specified
relative to the current filter's path.
*
Improvements to Customize Shell Extensions dialog.
*
New option for Grep filter to ignore empty matches.

What's New in TextPipe v8.4 - October 6, 2009
=============================================

*
New pre-compiling of replace expressions for faster replacing.
*
New 'Break on Change' option for captured variables makes debugging far
easier.
*
Revamped the simultaneous search/replace filter to work with lists of
300,000+ replacements. New Longest First option.
*
Copybook filter now copes with nested OCCURS where the nesting contains only
one field (special case).
*
Copybook filter now copes with OCCURS DEPENDING fields.
*
Copybook filter now outputs the header only once - very useful inside the
restrictions that are required for processing variable format files.
*
Copybook wizard - fixed disabling of trailing delimiter on last field.
*
New command line option /CLOSEONCOMPLETION - which closes the filter window
when a thread completes. This is typically used with a /Z command line file
and a large set of filters.
*
New option for Count Duplicates Filter to change the format of the results -
e.g. put tab between the count and the string, reverse the order etc.
*
Grep filter now has an option to match UTF-8 characters for Unicode text.
*
Convert to Word List filter now allows words with numbers such as Catch-22,
24-7, 5th, 7-Eleven etc.
*
Remove duplicate lines AND Count duplicate lines filters now do not require
that the duplicate lines be adjacent (i.e. that the list is pre-sorted).
However, it now has the restriction that it cannot speedily remove or show
duplicates from files LARGER than main memory.
*
Grep filter has now been changed so that it does NOT match empty lines.
*
TextPipe is now much faster when logging is disabled.
*
TextPipe no longer records output file names beyond a count of 10000. These
filenames are only used for .BAK file recovery, and they cause a performance
slow down when millions of files are being processed.
*
Fixed function for removing empty output files. The input file was being
deleted if the output file had zero size.
*
Fixed problem with duplicate data appearing in large sorts requiring merge
files with 2 or more merge stages.
*
Reverted handling of blank fields/blank lines to pre 8.3.5 handling. Pad to
Width and Add Left Margin filters cannot be used for blank fields (use an Add
Footer filter to give a blank field some width first).
*
Fixed bug in HTML Restriction filter where the action of the 'Include start
and end tag' option was inverted.

What's New in TextPipe v8.3.7 - July 17, 2009
=============================================

*
New Simultaneous search replace option for Filters\Search\Search and Replace
List Filter- use when your search and replace terms overlap each other e.g.
replace 1->7, 7->10, 10->1.
*
Copybook filter now recognises field names starting with numerics e.g. 14B-
DATE.
*
Copybook filter now copes with nested OCCURS, and numbers the generated field
names correctly.
*
New 'Copy to Clipboard' option when a hex dump is displayed from a field's
context menu.
*
New Output filter option to remove empty output files (those with a size of 0
KB).

What's New in TextPipe v8.3.6 - June 24, 2009
=============================================

*
Fixed performance of Trim option of mainframe copybook filter.

What's New in TextPipe v8.3.5 - May 5, 2009
===========================================

*
Fixed handling of blank fields when using Restriction Filter and a subfilter
such as Pad to Width, or Add Left Margin.
*
Fixed file modification detection when using subfilters of a Search/replace
or HTML Restriction filter.
*
Fixed memory leak that occurred when the Skip Binary files option was used.
*
Fixed VBScript/JScript code generated - missing end of line for
TPWindow.inputDeleteFiles.

What's New in TextPipe v8.3.4 - April 23, 2009
==============================================

*
Fixed bug with [Delete BAK files] button not being enabled.
*
Installer changes to support Vista better.

What's New in TextPipe v8.3.3 - March 27, 2009
==============================================

*
Fixed bug with Converting Excel spreadsheets to text when the output was set
to the clipboard.
*
If logging is enabled, a setting of 'Don't log to file if line count is less
than' = 0 now causes all lines to be logged, rather than none. To disable
logging, uncheck the logging flag. This primarily affects calling filters
from the TextPipe Engine.
*
The copying of all predefined global variables from the command line
(@name=value) now occurs no matter how the filter is called.
*
Enhanced Windows Explorer Context Menu handler to better handle huge sets of
files (20,000+).

What's New in TextPipe v8.3.2 - March 5, 2009
=============================================

*
Fixed bug in using UNC paths \\server\share etc - these conflicted with the
environment variable parser's desire to eliminate duplicate \\
*
Changed Directory scanning routines to allow files with any attributes.

What's New in TextPipe v8.3.1 - March 2, 2009
=============================================

*
Fixed bug in using external reference file for Replace->Search/replace list

What's New in TextPipe v8.3 - February 24, 2009
===============================================

*
Now recognizes the VALUE clause in mainframe copybooks.
*
EasyPatterns CSVfield, TABfield and Pipefield now handle unquoted fields
correctly.
*
Macros can now include environment variable values e.g. @ENV_OS inserts the
text 'Windows_NT' on an NT system.
*
Filenames specified in TextPipe can now make use of embedded environment
variables using e.g. %PROGRAMFILES%, %TEMP% syntax.
-	Search/replace list filter
-	Log filename
-	Execute filter input/output filename and command to run
-	Sound to play on completion, and default editor
-	Add Left/Right Margin, Header/Footer, Add Text Side by Side
-	Output filter new extension, new output folder and single output
filename
-	Merge and Link filters.
*
New Output Filter option for No Output - the filtering is used for its side
effect only.

What's New in TextPipe v8.2.3 - December 16, 2008
=================================================

*
Now recognizes the new Office 2007 format .xlsx (e.g. search/replace lists).
*
Fixed line count statistics displaying -1 for subfilters that are never
invoked.
*
Default is now to process compressed files (.zip, .docx, .xlsx, .pptx).
*
Restored play sound on completion for command line filters.

What's New in TextPipe v8.2.2 - December 15, 2008
=================================================

*
Removed playing a sound in non-interactive mode (from command line or from
COM).
*
Interactive error messages are now disabled when TextPipe is called from the
command line or from COM (when TextPipe.visible = false).

What's New in TextPipe v8.2.1 - December 9, 2008
================================================

*
Removed file copy on first run from Engine version.

What's New in TextPipe v8.2 - November 25, 2008
===============================================

*
New option to allow/prevent processing of files inside Compressed files such
as .ZIP, .DOCX, .XLSX and .PPTX.
*
[Copy lines to log file] button now also adds a log message to make the log
file easy to open.
*
Right-clicking the log results now shows an option to copy the log lines to
the clipboard.
*
Clearer warning for interactive logging ending.
*
Fixed EasyPatterns for CSVField, TabField and PipeField to avoid stack
overflows.
*
'Include the start and end tag' option of HTML filter was reversed.

What's New in TextPipe v8.1.12 - October 21, 2008
=================================================

*
'Output Filter\Open output on completion' now opens all output files, not
just the last one.
*
Better handling of copybook Comp3CanBeUnsigned option.
*
Now traps numeric conversion errors when sorting floating point values such
as 1e-9999.
*
Word concordance now has correct default for the output format, and the found
word is now correctly positioned in the context, and capitalized.

What's New in TextPipe v8.1.11 - October 6, 2008
================================================

*
Fixed buffer underrun problem with search/replace filter, where maximum match
sizes larger than 4KB were not correctly performed.

What's New in TextPipe v8.1.10 - September 8, 2008
==================================================

*
New 'Paste File List from Clipboard' button on Files To Process tab.
*
Files to Process grid is now refreshed after loading a File List.
*
Embedded Help is now loaded correctly from the application folder.
*
URLs can now be inserted or loaded from a File List without being interpreted
incorrectly as filenames.

What's New in TextPipe v8.1.9 - August 26, 2008
===============================================

*
Fixed flushing of T-filter child filters.

What's New in TextPipe v8.1.8 - August 25, 2008
===============================================

*
Added new 'T-filter' to copy the data and process it in a different branch
without affecting the main branch.
*
Changed Unicode filter Unicode to Escaped ASCII to also convert Latin (128-
255) characters to escaped \Uxxxx format.
*
Improved right-clicking in filter tree to first select the node that is
right-clicked on.
*
Improved handling of invalid zip files.

What's New in TextPipe v8.1.7 - July 19, 2008
=============================================

*
Fixed problem loading replace filters from versions 7.8.3 and older with \@
or \$ in the replacement text.

What's New in TextPipe v8.1.6 - July 9, 2008
============================================

*
Added new filter to Restrict to UTF-8 files.

What's New in TextPipe v8.1.5 - July 2, 2008
============================================

*
Fixed bug with named parameters in search/replaces - if a subexpression was
not captured then any remaining named parameters would have no value and
corresponding global variables would not be set.

What's New in TextPipe v8.1.4 - June 24, 2008
=============================================

*
Clipboard output now shows a non-zero size.
*
Statistics are now reset correctly when a filter has subfiters.
*
Copy to log button now correctly re-enabled when a job starts for the second
time.

What's New in TextPipe v8.1.3 - June 16, 2008
=============================================

*
Improved internal patterns used by the HTML filter to prevent stack overflow
errors.

What's New in TextPipe v8.1.2 - May 27, 2008
============================================

*
Fixed bug where /Q and /MINIMIZED were not recognised when placed in a /Z=
command file.
*
Fixed bug where subfilters of Restrict to Matching lines were not called when
a count of matching lines was being output.
*
Fixed discrepancies between docs, TextPipe and TextPipe Engine for
addSecondaryOutputFilter() function, output options and for filenames in the
File Grid.

What's New in TextPipe v8.1.1 - April 29, 2008
==============================================

*
Fixed error using a Search/Replace action of 'Send variable X to subfilter'
when the previous replacement text had been set to '$0'.

What's New in TextPipe v8.1 - April 21, 2008
============================================

*
Improved handling of non-DOS EOL files - TextPipe now checks if your filter
has an EOL Conversion filter before offering to add one for you.
*
Link filters can now use a path relative to the current filters path
(previously this was relative to the TextPipe installation path). This makes
them far easier to link to.
*
The Capture Filter now allows variable names containing digits.
*
Globals can now be set on the command line using the syntax @name=value with
a space between each set of values.

What's New in TextPipe v8.0.9 - March 25, 2008
==============================================

*
New extended logging for Remove Blank Lines, Remove Blanks from Start/End of
Line.

What's New in TextPipe v8.0.8 - March 18, 2008
==============================================

*
New notice level message if MS Word or Excel files are processed using the
evaluation version.
*
If Remove/Restrict Delimited Field filters remove fields from the end of the
line, TextPipe now removes any hanging delimiters.
*
The Convert Excel files to Text filter now converts all sheets, not just the
first one.

What's New in TextPipe v8.0.7 - March 5, 2008
=============================================

*
Zoned decimal conversions now allow positive and negative signs.

What's New in TextPipe v8.0.6 - February 20, 2008
=================================================

*
When run from the command line, TextPipe now does not show a dialog on errors
such as a filter file could not be loaded. An exit code is set and the log
file written instead.
*
Fixed a bug where the Analyse function on the Files Grid did not return a
result.

What's New in TextPipe v8.0.5 - February 14, 2008
=================================================

*
Fixed a bug where the Copybook Filter used the wrong parser if PIC was
preceded by a tab instead of a space.
*
Copybook Filter now copes with 'PIC GROUP' syntax.

What's New in TextPipe v8.0.4 - February 13, 2008
=================================================

*
Capture filter can now resets at the beginning of the file, at the beginning
of each region or both.

What's New in TextPipe v8.0.3 - February 4, 2008
================================================

*
Right-clicking a filter now shows a 'Help on this filter' link which quickly
takes you to the help file without needing to add the filter. Using
Help\Embedded Help also constantly updates to show you help on the current
filter.
*
The Status tab is now displayed by default when filters are run from the
command-line or from Windows Explorer.
*
Mainframe filters now insert implied decimals correctly when the fields are
blank padded.
*
Folder overload prevention for the split filter now triggers at 1000 files
(was 10000).

What's New in TextPipe v8.0.2 - January 9, 2008
===============================================

*
Now warns if the user tries to use the Trial Run area to process a Mainframe
Copybook (a file must be used).
*
Task Menu filters now open a status window so that progress and/or errors can
be reviewed.
*
The evaluation version (even an expired version) can now be used to check if
a filter is compatible with the Pro, Standard or Lite version.
*
The Restrict to Filename filter (used to rename files on the fly) now skips
processing filenames inside zip/docx/xlsx/pptx files.

What's New in TextPipe v8.0.1 - October 30, 2007
================================================

*
Fixed implied decimal inserted in wrong position for zoned decimal fields.

What's New in TextPipe v8 - October 24, 2007
============================================

*
The Status Window now forms a new tab of the Filter Window - making it easier
to refer to results. The results log is now also far easier to read.
*
The Mainframe Copybook Filter (new) and Wizard (deprecated)
- now handle OCCURS without TIMES when the repeating field is an unnamed PIC.
- now handle masks like PIC 9999/99/99 - these are treated just as an Alpha
field, in this case, PIC X(8). No mask substation is performed.
*
The Mainframe Copybook Filter now generates numbered field names when the
fields are inside a repeated group. This wizard does not do this, but the
field header can be manually edited.
*
Improved the EasyPatterns CSVfield, TABfield, PipeField, HTMLtag,
HTMLstartTag, HTMLendTag, QuotedString so that they are much faster.

What's New in TextPipe v7.9.11 - October 24, 2007
=================================================

*
Fixed bug with Search/Replace in Exact mode, if the Action was set to
'Remove', the text was not actually removed.
*
Added collection and logging of statistics to Lite and Standard editions.

What's New in TextPipe v7.9.10 - October 19, 2007
=================================================

*
Log filenames are now displayed with an absolute path to make them easier to
locate. When [Copy Lines to Log File] is clicked in the Status Window, now
ALL lines are copied.
*
Fixed handling of invalid .zip files.
*
Convert MS Word/Excel documents to text - better handling of MS Word/Excel
failures.
*
Improved status dialog with single tab display.
*
Log files now combine date and time into single field.

What's New in TextPipe v7.9.9 - October 8, 2007
===============================================

*
Fixed output of implied decimal values for Mainframe Copybook conversions.

What's New in TextPipe v7.9.8 - September 27, 2007
==================================================

*
Fixed binary file detection logic.

What's New in TextPipe v7.9.7c - September 18, 2007
===================================================

*
Fixed handling of last file in compressed Office 2007 formats (docx, xlsx,
pptx).

What's New in TextPipe v7.9.7b - September 12, 2007
===================================================

*
Fixed script generation for addUnicodeConversionFilter2.
*
Fixed COM function TextPipeFilter.compiledFilterStartJob.
*
Grep functions can now match blank lines using the pattern ^$.
*
Fixed Copybook Filter choosing incorrect parser if 'pic' used rather than
'PIC'. Copybook Filter now accepts field names that include spaces. Parse
tree output now includes periods so that it is still a valid copybook. Fixed
field width for P (Packed) type when using alternate style copybooks.

What's New in TextPipe v7.9.7 - September 4, 2007
=================================================

*
The File Input's Binary detection now correctly identifies Unicode files, Zip
files and Microsoft Office 2007 file formats (.DOCX, .XLSX, .PPTX) as non-
binary.

What's New in TextPipe v7.9.6a - August 29, 2007
================================================

*
Debug Mode of Copybook Filter now shows field length and start and stop
columns. Copybook Filter now handles OCCURS after the PIC definition.
*
New internal editor for reviewing text files.

What's New in TextPipe v7.9.6 - August 24, 2007
===============================================

*
Zip files - now processes and updates data stored inside compressed files
such as .zip, and Microsoft Office 2007 file formats such .DOCX, .XLSX, .PPTX
etc.
*
Word conversion filter now clears previous results in case of error, and
disables macros during run.
*
Removed 64k limitation on string data saved to registry, fixed error with
string data over 64k not being allowed.

What's New in TextPipe v7.9.5 - August 10, 2007
===============================================

*
Fixed Width Wizard now accepts both
          and
     -
formats.
*
Now uses updated pattern match library to resolve regex problems.

What's New in TextPipe v7.9.4 - August 1, 2007
==============================================

*
New status window option to Delete backup files (in addition to restoring
backup files). Status window now warns you if you attempt to close it before
dealing with backup files.
*
Better warning message for invalid file patterns in the Files to Process
grid.
*
Copybook REDEFINE clause now allows an optional trailing period.

What's New in TextPipe v7.9.3 - June 22, 2007
=============================================

*
New Word Concordance filter under Filters\Convert.
*
New filter Restrict to Blocks of Lines - useful for adding text after every
XX lines.
*
Reinstated Filters menu for accessibility, and File Menu items Link to
Filter, Insert a Filter, Search for a filter.
*
Refined the pattern generated by filter Restrict to Between Tags.
*
Script filter now works with PerlScript engine - all function validation is
disabled.

What's New in TextPipe v7.9.2 - June 12, 2007
=============================================

*
Added semi-colon (;) as an output format for the Mainframe Copybook Wizard
and Fixed Width Wizard.
*
Added error for blank text in the Mainframe Copybook Filter.
*
Fixed incorrect length detection for signed zoned decimal in the Mainframe
Copybook Wizard (the Copybook Filter used the correct result).
*
Reinstated wizard on startup.

What's New in TextPipe v7.9.1 - May 31, 2007
============================================

*
Fixed pattern generated by filter Restrict to Between Tags.
*
The default filter Normal.fll is now loaded from the Documents folder
regardless of which folder TextPipe is started from.
*
C:\ is now not used as a temporary folder, instead it is requested from
Windows.
*
Maps are now retrieved from the Documents folder.
*
Code page functions are now unavailable under Windows NT.
*
New prompt for language on first run.

What's New in TextPipe v7.9 - May 28, 2007
==========================================

*
New Multi-Processor support (in Server version) with Filters\Special\Switch
CPUs to direct filtering to different CPUs. This enables massive performance
enhancements for large data crunching tasks.
*
Vista install support.
*
Now detects Unix files being loaded into Trial Run area, and warns the user
and offers to add an End-Of-Line conversion filter.
*
Mainframe Copybook Wizard - fixed incorrect implied decimal.
- has been completely re-written as a filter. The parser now shows the line
number of any parse error, handles INDEXED BY clauses, logs warning if
invalid record at end of file, XML output option.
- now handles non-PIC style copybooks
*
TextPipe now highlights the filter in error correctly, taking into account
comment filters.
*
New macro @filterFilename to output the filter filename.
TextPipe.filterFilename is also available in the scripting filter.
*
Comment filters are now shown in green italics to make the filter list more
readable.
*
Fixed pattern generated by filter Restrict to Between Tags.
*
An inability to log messages to the log file now causes a Windows Event Log.
*
Tightened up the Analyse File Format function to correctly identify BOM and
DOS End-Of-Lines in files.
*
Fixed intermittent /Q Exit On Completion switch. This only occurred on fast
CPUs when a new processing thread started and finished before (!) the main
thread processed the /Q command.
*
Fixed EasyPattern bracket optimization. Now shows the line number of any
error, and fixed [PhoneNumber] pattern.
*
Fixed Perl/EasyPattern exception with named captured variables and Prompt on
Replace was turned on, when the named captured variable did not match
anything.
*
Removed remaining memory Info log item on Merge Sort.

What's New in TextPipe v7.8.13
==============================

*
Mainframe Zoned Decimal expansion now allows negative to be coded with '2' or
'5'.
*
Comment and Scripting filters now get display of special characters turned
off by default.
*
Mainframe Wizard now removes the trailing delimiter on the last field.

What's New in TextPipe v7.8.12
==============================

*
Explorer Context Menu now shows the filename without the extension.
*
Convert Excel to CSV filter now does not use a TEMP file in C:\
*
Mainframe Copybook Wizard now
- correctly discards INDEXED BY statements.
- removes the FILLER header associated with removed FILLER fields.
- VALUE lines are now ignored rather than causing a warning.
*
Windows from TextPipe.Alert calls now allow the filter to be cancelled.

What's New in TextPipe v7.8.11
==============================

*
Fixed bug with handling of Replace Action of 'Send $x to subfilter'.
*
Fixed bug with HTML filter removing the Restricted text.

What's New in TextPipe v7.8.10
==============================

*
New 'Action' field for Search/Replaces, with the default being just
'Replace', as before. The new field allows the matched value to be Removed,
to be sent to subfilters, or send just part of the matched value, such as $1-
$9. It also allows NON-matching text to be subfiltered, making it very easy
to process text outside bracketed expressions like
*
New Output Filter options:
- Only Output Modified Files has been extended to allow the original files to
be deleted if they have been changed. This is very useful for repeated
processing of files to different output folders.
- Ignore Initial Folder - when the folder structure is being maintained, this
allows the start folder to be removed from the destination folder name.
*
EasyPatterns now support named capturing using the syntax:
  [ capture( ... ) as 'VarName' ]
*
Optimized the Head/Tail filter when Including Bytes from the End of a file.
*
The Filters Remove Lines From List and Extract Lines From List now do not
recognize special characters such as \. This is so that lists of filenames
with backslashes can be pasted in.
*
File Grid 'Ignore' option renamed to 'Disable'. File Grid now allows files to
be excluded by perl regex pattern or EasyPattern.
*
Replace List filter now allows Find/Replace strings to be Swapped, and the
list to be reversed - very useful if you apply changes to your data and then
reverse them at the end of the filter list.
*
Fixed bug with the Restrict to Filenames Matching Pattern filter
-	the filename was not being passed to subfilters, so double-level
filename restrictions did not work
-	the patterns were incorrectly unescaped before being parsed, so
patterns with backslashes were not correctly interpreted.
*
If too much data is sent to the Clipboard, TextPipe now traps the error and
limits the text to 10MB.
*
Fixed bug with Restrict to Lines in Pattern list - the patterns were
incorrectly unescaped before being parsed, so patterns with backslashes were
not correctly interpreted.
*
Added scripting functions TextPipe.fullOutputFilename, TextPipe.FilterWindow,
TextPipe.Application.
*
Added COM functions FilterWindow.Application, FilterWindow.outputFileCount,
FilterWindow.outputFile() and FilterWindow.outputFileTask().
*
Now warns against CSV or Tab data containing ASCII 26 when converting to XML
(remove or change using a map or Search/Replace filter).
*
Fixed EasyPattern for [EmailAddress].
*
Updated Extract\Email addresses filter.
*
Fixed bug in Match Between HTML tags - Match Case was being set to On rather
than using the displayed setting.

What's New in TextPipe v7.8.9
=============================

*
Increased maximum stack size to eliminate Stack Overflow errors.
*
Fixed bug in Capture Text filter where the new captured value was only set at
the end of a file.

What's New in TextPipe v7.8.8
=============================

*
Processing is now suspended when you hit [Cancel].
*
Added TextPipe.exe path to Run... path.
*
Fixed bug - the Mainframe Copybook Wizard no longer tries to replace
delimiters in the input file. It was replacing delimiters even inside packed
decimal fields.

What's New in TextPipe v7.8.7
=============================

*
Fixed parsing of replace expressions including \x24 (a $ sign).
*
Mainframe Copybook Wizard new options - Remove Filler fields, Include header.
The Wizard now records option settings in the Comment output.

What's New in TextPipe v7.8.6
=============================

*
Fixed sort function not working when start column <> 1 and length < 4096.
*
Extract email addresses now ignores '&'.

What's New in TextPipe v7.8.5
=============================

*
Loading a search/replace list from Microsoft Excel now allows Unicode values.
*
The grep filter now always reports the output file as being 'modified',
preventing confusing behaviour with the 'Only output modified files' Output
Filter option. A change of filename from input to output now also results in
a file being written, regardless of any changes.

What's New in TextPipe v7.8.4
=============================

*
Mainframe Copybook Wizard now handles PIC fields both before and after OCCURS
xx TIMES for single field OCCURS statements.
*
Optimized tree updating as field data is changed.
*
Changed install paths for Windows Vista.

What's New in TextPipe v7.8.3
=============================

*
New internal function TextPipe.finishedWithFile() to halt any further reading
of the current input file.
*
New TextPipe.Application function isBinaryFile() to detect if a file is
binary or not.
*
Updated memo editor and pattern matching engine to fix bugs in third-party
components.
*
Fixed EasyPattern handling of EBCDIC() function.
*
Fixed hex dump filter outputting the last line twice.
*
Fixed handling of Compress to EBCDIC Packed Decimal.
*
Fixed bug converting old replace patterns of form $1$2$3 to new form of
$1$$2$$3.

What's New in TextPipe v7.8.2
=============================

*
New filter to Add a Line at a given line number.
*
Fixed bug with EasyPatterns not returning all of a captured expression.
*
Unicode conversion filter now makes use of the 'error' character.
*
Mainframe copybook filter now copes with INDEXED BY statements, and better
handles nested OCCURS.

What's New in TextPipe v7.8.1
=============================

*
Escaped Unicode now outputs \uXXXX instead of \u+XXXX.

What's New in TextPipe v7.8
===========================

*
New option to show special characters such as spaces, tabs and line feeds.
*
New option - triple-clicking a line selects the whole line.
*
New filter to convert Filters\Convert\UTF16-LE to Escaped Unicode (ASCII +
UCN \U) e.g. \u402d\u403d
*
The file list can now contain http://, ftp:// and gopher:// urls to download
directly.
*
The user is now given the option to manually start Word or Excel if they are
not running when TextPipe needs them (with the Convert Word/Excel Document to
Text filters).
*
Pattern matches can now name captured variables for use in the replacement
text e.g. searching for (?P\d+) would then allow the macro @dollars
to be used in the replacement, and also as a global anywhere else in the
filter list.
*
Far more complex patterns can be processed, with support for up to 99
captured variables, from $1-$99.
*
New 'File\Send to...' function for filter lists.
*
New Fixed Width to Delimited Output wizard - makes converting fixed width
files to CSV, Tab simple.
*
Mainframe Copybook Wizard now allows underscores (_) and single quotes (') in
field names in  'XXX_XXXX 4 AN 1-4'-type copybooks.
Wizard now removes the content of fields called FILLER*.
Wizard now copes with repeated single-fields, as well as GROUPED OCCURS.
EBCDIC packed decimal fields can now have either a leading sign, trailing
sign or no sign.
New option to trim leading spaces and zeroes from fields.
New option to only expand numeric fields - Alpha fields are left unchanged.
This is useful when the target application can accept EBCDIC data but can't
handle packed decimals.
New option to allow COMP-3 to be unsigned.
Wizard can now cope with removing line numbers and commented copybook lines
(those with a '*' in column 7).
Wizard can now cope with field definitions spread over multiple lines.
Wizard now remembers settings between uses in a single session.
Expand Binary Number to EBCDIC filter now outputs a fixed width result.
Expand Unsigned EBCDIC filters now output a space for invalid digits A-F.
*
EasyPatterns now produce optimized perl regex output. Added EasyPattern
EBCDIC('') for matching EBCDIC values. Fixed parsing error with ASCII()
function. Now handles back references to groups 1-99. Fixed DayNumber
insisting on leading zero. EBCDIC() and ASCII() functions can now be used in
conjunction with not ... as character classes.
*
Hex displays now all use '.' to replace space.
Query replace window now shows hex offsets for Hex Dump display.
*
Convert Word Document to Text/Convert Excel Spreadsheets to Text now support
Unicode output as an option. Follow this with Convert UTF-16 to UTF-8 to
process these files using normal filters.
*
New File Shred filter to completely overwrite file contents. Run repeatedly
for more shredding.
*
Unicode property tables for pattern matching updated to Unicode 4.1.0, and
matching engine updated to PCRE 6.4.
*
Split filter now can split at a pattern.
New Split Filter option to avoid overloading directories. When more than
10000 files are created, TextPipe starts a new folder.
Fixed bug where \\ could appear in split output filename.
*
Filter list tree now auto-scrolls.
Filter list tree now allows mass enable/disable of filters with select and
right-click.
*
New Filters to Add File Headers/Footers. Unlike the existing Header/Footer
filters, these filters only output once per file even when included inside a
restriction.
*
Now can drag and drop filters between filter lists in different windows.
*
Filters Add Header/Footer/Left Margin/Right Margin now always check for
@macros embedded in the text, regardless of whether the text comes from the
memo or an external file.
*
Fixed bug with search/replaces where an internal conversion from ansi to
Unicode and back could result in high-ansi characters not being mapped
correctly back to their original characters. In particular, the ansi (tm)
symbol mapped back to TWO characters.

What's New in TextPipe v7.7.4
=============================

*
@macros are now allowed in the replace field of a search/replace.
Note: If you load a filter from an older version, '@' will be escaped to
'@@'. The Confirm Replace dialog now shows the values of all global and user-
defined @macros.
*
Warning for Excel or Word documents now disabled.
*
Fixed Seller's Edit Distance search/replace not working.
*
Fixed bug in Mainframe Copybook Wizard for length of signed Packed Decimal
fields.

What's New in TextPipe v7.7.3
=============================

*
No longer deletes current output file when processing is cancelled and the
file is being appended to.

What's New in TextPipe v7.7.2
=============================

*
TextPipe now works with DataPipe successfully.

What's New in TextPipe v7.7.1
=============================

*
To prevent accidental overwriting of files, the default path for the Output
Filter's Single File Output, and the Merge Filter path are now set to blank.
*
Removed the support for quoted tab data. Tab data does not need quotes!
*
Fixed error with deleting filters.
*
Fixed error entering custom delimiter when '\t' was entered.

What's New in TextPipe v7.7
===========================

*
Ultra-fast sorting of huge files (100MB in less than 30 seconds); control
over maximum memory used - for shared servers.
*
Convert Excel spreadsheets to CSV format.
*
Control which filters are applied to which file types with the new Filename
Restriction filter - apply filters to say *.doc files, or *.xls etc.
*
Handle more types of Mainframe data than ever before, with the improved
Mainframe Copybook Wizard:
- discarding line numbers from pasted copybooks
- now handles null packed and zoned numeric fields
- now outputs invalid digits in packed or zoned numerics as their hex
equivalents
- now adds filters to trim leading and trailing space from fields, and
leading '+' signs from numerics
- can now choose output format between Fixed, CSV, Tab- and Pipe-delimited
- now can add leading zeroes to numeric fields where the implied position is
greater than the field width
- output format now includes a record header, which makes it far easier to
validate the output file and import the file into a database
- new filters for unpacking odd/even-length unsigned packed numerics, and for
decoding overlapped unsigned packed numerics
- delimiters are removed from the input file first to prevent extra fields
being created.
*
Easily Extract Text Between Delimiters with the new Web-based Filter Wizard.
*
Review results easily
 - New Output Filter option to open the output file on completion.
 - Open output files easily with clickable links in the Status Dialog.
*
Set the Error Character for Unicode and Code Page conversions when there is
no appropriate character in the target encoding.
*
For precision automation, control TextPipe's pause before exit with the new
/Q=millisecond option.
*
Accept pasted Unix text in the Search/replace filter.
*
Easily match day names, day numbers and arbitrary characters using the new
EasyPatterns [dayname], [daynumber] and [ascii()] function which inserts
arbitrary ASCII codes in hex (eg $ff) or decimal (e.g. 255) format.
EasyPatterns now accept counts and quantities in either hex or decimal
format.
*
Numbers lines across files, with the 'Don't Reset At End Of File' option of
the Line Numbering filter.
*
Count occurrences of all lines, with the 'Include Counts Of 1' option of the
Count Duplicate Lines filter.
*
Debug the output files of External programs using the 'Remove Output Files on
Completion' option of the Execute Program Filter.
*
Store arbitrary text in the Scratch Pad - new Edit\Preference option.
*
Date and time macros now return consistent results across different
localities -
  @time - Inserts the time in the format HHMMSS
  @date - Inserts the date in the format YYYYMMDD
  @datetime - Inserts the date and time in the format YYYYMMDDHHMMSS
*
When the right-click menu 'Save' option is selected, TextPipe now saves the
text exactly as is without interpreting special characters.
*
TextPipe now prompts if you try and exit while a thread is running, and asks
for confirmation if you click [Cancel].
*
When a list of perl patterns is loaded or used, TextPipe now shows the line
number of any error when a pattern parsing error occurs.
*
The grep form's context menu now displays EasyPatterns or perl patterns
according to the pattern type selection.
*
When new filters are added or duplicated they become selected. New filters
are now added with blank fields, and for the Search/Replace filter, the extra
options are set to defaults. After deleting a filter, the next filter is now
selected.
*
The Options panel of the filter tree now scrolls to display buttons that are
not visible.
*
Better handling of files that are missing or open in another process.
*
Fixed Range Check error with optimization of 1:1 maps.
*
Fixed Single File Output - didn't force creation of folders for output file.
*
Fixed Windows Explorer shell extension - was starting the filter one row down
from the correct one.
*
Fixed bug where Extract URLs filter would miss a URL on the very last line.
*
Fixed merge sorting bug (only for very large files) where processing never
ended, and duplicates were not removed.

What's New in TextPipe v7.6.3
=============================

*
New macro @fullOutputFilename. May be blank if the clipboard is being
processed.
*
Fixed deadlock between threads that was slowing down the processing when the
TextPipe.Application object was called from code.
*
Fixed Windows Explorer Content Menu not appearing.

What's New in TextPipe v7.6.2
=============================

*
Fixed bug with grep filters where Match Case option was not recognised.
*
Standardized help text and labels for grep filters.
*
Fixed bug with conversions between ANSI and ASCII.
*
Fixed bug with delimited field parser when it encountered a ' or " in the
middle of the field.
*
Fixed bug with clipboard button not being acted on correctly.

What's New in TextPipe v7.6.1
=============================

*
New filters to Restrict to ANSI files, and Restrict to Unicode UTF16 files
and Unicode UTF32 files. These are exceptionally handy for only converting
ANSI files to Unicode, or only converting Unicode files to ANSI.
*
New filter to convert PDF documents to text format.
*
Menu re-organization: Email menu replaced by new Extract menu, encodings
moved to Convert menu. 'Retain...' items renamed to 'Extract...'.
*
New macros @randomdigit and @randomletter - very useful for generating test
data or for generating random sort keys.
*
New Explorer extension option to Analyse files - detect if they are Unicode,
ANSI, etc.
*
Reverse File performance improvement.

What's New in TextPipe v7.6
===========================

*
New Unicode filters for Upper case, Lower case and Title case.
*
Speed improvements for handling very large files.
*
New Expand All/Collapse All menu on All Filters view.
*
New scripting function to build filters to convert a Mainframe Copybook.
Copybook wizard now inserts a warning for each unidentified field, making
incorrect fields easy to identify.
*
New warning message if no files were found to process.
*
Filters\Convert\Numeric HTML entities To Text filter now converts output
characters to UTF-8. The resulting text can then be converted from UTF-8 to
UTF-16 for display.
*
Menu items for common grep and conversion filters in the Wizard menu. Wizard
menu now is always expanded.
*
Memo and text fields now use the fixed width font.
*
Unicode and code page conversions now do not load the entire file into
memory, they convert the file progressively.
*
Updated Simplified Chinese translation.
*
Help text added to the Unicode/Code page conversion filter.
*
Moved Hex and HTTP encode/decode to the Convert menu.
*
Fixed pasting of text into a memo's Find window.

What's New in TextPipe v7.5.1
=============================

*
Drag and drop now prevents nodes being dropped on filters that do not support
subfilters.
*
Unicode replacement now reverses the order of Match Case to be correct.
*
Reverted help system to old style.

What's New in TextPipe v7.5
===========================

*
New Restrict to lines from list/lines NOT from list and
Restrict to lines matching pattern list/lines NOT matching pattern list.
New Match Case option for these filters too.
*
EasyPatterns for Month and MonthName now do not automatically capture.
*
Mainframe Copybook Wizard now copes with OCCURS where the number of
repetitions is fixed.
*
New filter to convert numeric HTML entities to text e.g. ·
. New
maps to convert text to decimal or hex HTML entities.
*
/CLEARFILTERS now removes all filters except the Log, Input and Output
filters. This retains the default settings from normal.fll
*
/EXTENSION can now be set to blank to disable changing the extension.
*
Fixed Access Violation processing empty result sets.
*
Fixed load file list function.

What's New in TextPipe v7.4.3
=============================

*
Now shows all available filters in a separate tree - these can be drag and
dropped or double-clicked to move them into the filter tree.
*
Fix to Mainframe copybook filter to allow it to recognise field names that
included digits followed by ' A'.

What's New in TextPipe v7.4.2
=============================

*
Fixed bug with Unicode exact match and Unicode pattern match where extra
characters were being output.

What's New in TextPipe v7.4.1
=============================

*
Fixed bug with Unicode exact match and Unicode pattern match where search
terms were not found.

What's New in TextPipe v7.4
===========================

*
Mainframe copybook wizard is now more flexible in the data it matches, and
handles COMP-3 field types, and handles numeric fields with purely decimals
(eg V999).
*
New filters for code page conversion.
*
New filters for Unicode Compose and Decompose:
  NFC - Canonical Decomposition, followed by Canonical Composition
  NFD - Canonical Decomposition
  NFKD - Compatibility Decomposition
  NFKC - Compatibility Decomposition followed by Canonical Composition
  And Canonical Compose.
*
Now supports Unicode command line parameters.
*
Now supports single quoted CSV data, quoted tab data, and backslashes in
quoted CSV or Tab data. Existing text delimiters are now preserved on
conversion between CSV and Tab formats. The EasyPattern [tabfield] can now
match quoted tab data.
*
The Restrict to HTML filter can now set the maximum match size - to match
very large sections of HTML or XML.
*
Clarified examples in perl pattern right-click menus.
*
Speed optimization for interactive search/replace when [Replace Job] chosen.
*
When files are dropped, TextPipe defaults files to 'no subfolders'.
*
Simplified the Trial Run button - now it only performs a trial run, and does
not toggle back to the Filters to Apply tab.
*
XSLT filter now correctly flags modified text - so output files get created
when the 'Only Output Modified Files' flag is set. XSLT filter now reports
the error code and error reason to the log if an error occurs.
*
Run External Program filter now reports in the log if the external program
has a non-zero exit code. The output file handle is now released on error.
*
The Unicode Map filter's 'Replace With Value' string now supports special
characters.
*
Encode Hex filter now does not add line feeds.
*
Right-click option Show Hex Dump now checks if the underlying field is
Unicode or not before displaying data. A toggle between UTF16-LE and UTF16-BE
is also available.
*
Reduced memory requirements of Database filter when SELECTing data sets.
*
Fixed database filter's handling of VarBinary (BLOB) data.
*
Database filter now handles subfilters correctly for encoding VarBinary data.
*
Fixed bug when referencing a CSV or Tab search/replace list where the
replacement text was missing. A new error message now provides the line
number of any errors.
*
Fixed bug with inserting captured text as an HTML table row, and added the
ability to insert as an XML record.
*
Fixed bug with HTML help crashing TextPipe on exit.
*
Fixed bug with Input Filter, Prompt On Binary files setting, if 'No To All'
was chosen TextPipe skipped ALL remaining files, not just binary files.
*
Fixed Unicode display of extended information in the Prompt on Replace
dialog. Note: Unicode patterns do not support capturing- only $0 works.
*
Fixed Map filter not detecting Reset and Clear changes.
*
Fixed help file opening twice when F1 pressed.
*
Fixed mismatch of 16 and 32-bit Unicode strings when converting from ANSI or
ASCII/OEM to Unicode and between code pages.

What's New in TextPipe v7.3.1
=============================

*
Restored registry settings required by Offline Explorer.
*
Changed default for re-loading previous filter to Off.
*
Better exception handling for XSLT filter.

What's New in TextPipe v7.3
===========================

*
New Mainframe Copybook wizard makes it incredibly easy to build a filter to
process mainframe data. Handles packed, zoned, binary and display field
types.
*
New filters to Expand Binary Number To ASCII and Expand Binary Number To
EBCDIC.
*
Global dialogs now make use of popup menus to insert special characters etc.

What's New in TextPipe v7.2.1
=============================

*
New filter to capture text to a global variable. This is much faster than
using the Script filter to capture and then output text - which is the most
common use of the Script filter.
*
New grid component for the Map filter - now supports far better range and
mouse selection.
*
New option to automatically reopen the last filter.
*
The Replace Query dialog now has an extra tab to show a hex dump of the found
text. This can be useful for debugging.

What's New in TextPipe v7.2
===========================

*
Right-click menu for Replace field now provides easy insertion of captured
data as Tab or CSV-delimited fields or HTML.
*
The Prompt On Replace dialog now displays the length of each captured
expression.
*
EasyPatterns 1.9.1 - added [PipeField] and [Pipe] keywords.
*
Fixed error when manually choosing the Skip option to not replace text in a
DBCS file - more than one character was added. Only occurs under DBCS
versions of Windows (eg Chinese).
*
New error message when there are too many capturing expressions in a pattern.
*
Fixed bug with 'First row is header row' not being recognised for processing
delimited data with embedded carriage returns.

What's New in TextPipe v7.1.7
=============================

*
Added short cut menu items for converting between Unicode/ANSI and
Unicode/ASCII.
*
Added line number and offset properties to the TextPipe object available in
the scripting filter.
*
Map filters now have a comment property to describe what they are doing.
*
EasyPatterns 1.9 - added keywords for [number, numbers, leftAngle,
rightAngle, lessThan, greaterThan ]
*
Fixed restriction filters not supporting Secondary Output filters properly.
*
Fixed exact match filter not properly supporting Extract option, and not
correctly handling text when 'Skip' was chosen.

What's New in TextPipe v7.1.6
=============================

*
Search/replace lists now detect inserts, deletes and moves as changes. The
order of moved rows is now retained.
*
Fixed restriction filters not supporting Secondary Output filters properly.
*
Fixed EP keywords for [CRLF], [DOSNewLine], [EmailAddress] and [Hyperlink]
*
Fixed width font for Trial Run Input/Output is now correctly loaded.

What's New in TextPipe v7.1.5
=============================

*
EasyPatterns now allow repetition to be specified using the forms [2..3 ...]
and [2-3 ...] in addition to [2 to 3 ...].
*
Right-click menu item Escape Special Characters now works for EasyPattern
fields.
*
Fixed format string error on filter export.
*
Fixed Search/Replace list -caused an error when it was opened.
*
Fixed Remove Byte Order Mark (BOM) filter when handling UTF-8 files.

What's New in TextPipe v7.1.4
=============================

*
Added right-click menu to EasyPattern fields to make designing patterns
easier.
*
Added Line, Column and Length indicators to Search/replace filter.
*
A warning is now shown for trailing whitespace in the search text, which
often results in the text not being found.
*
Added 5 second countdown before TextPipe exits due to /Q command line
parameter.
*
Greatly improved performance for massive numbers of files- TextPipe no longer
checks for duplicate filenames.
*
Fixed EasyPattern for [month,monthnumber,day,hour] so that they match the
long version of the pattern first, then try the shorter variations.
*
Fixed range check errors with Compress packed/zoned decimal filters.

What's New in TextPipe v7.1.3
=============================

*
Fixed EasyPattern for [word] - previously matched non-word characters.
*
Fixed EasyPattern - '.' was not escaped - now fixed.
*
The Prompt On Replace dialog now shows the perl equivalent to an EasyPattern,
for verification purposes.

What's New in TextPipe v7.1.2
=============================

*
Fixed access violation with changes to the Restrict to Bytes filter.

What's New in TextPipe v7.1.1
=============================

*
Improved Restrict to Bytes (Byte x to End-y) to handle unlimited amounts of
data

What's New in TextPipe v7.1
===========================

*
You can now set the field delimiter for the Restrict To Fields, Copy Fields,
Move Fields and Delete Fields filters. This opens the way for semi-colon-
delimited, #-delimited, !-delimited, ^-delimited and other bizarre data
formats.
*
Added two new filters for compressing mainframe EBCDIC data to Packed Decimal
and Zoned Decimal formats.
*
The Split filter can now set the first filename number.
*
Added new Insert Bytes filter to complement the Insert Columns filter. This
new filter disregards line feeds when it adds text, and is very useful for
inserting commas into EBCDIC text as the text is converted.
*
New documentation links in the Filter Menus.
*
Synchronized scrolling is now properly restored when a filter is loaded.
*
EasyPattern fixes to align with the EasyPattern documentation
- [symbol, symbols] - removed the '!'
- [space, spaces] changed to just matched ASCII 32, not tabs, line feeds etc
- [horizontalwhitespace, hspace] - now only match space and tab
- [verticalwhitespace, vspace, worddelimiter, linedelimiter,
paragraphdelimiter] - now match formfeed
- [anybracket, anybrackets] - now match { and }
*
EasyPatterns now will NOT attempt to combine OR'd negated and non-negated
character sets, such as [ not lineChar or 'a' ]
*
HTML filter now matches tags when the tag name is immediately followed by a
tab or newline, not just a space.

What's New in TextPipe v7.0.8
=============================

*
Fixed EasyPattern handling of vertical tab (\v), which affects the keywords
verticaltab, verticalwhitespace, vspace, worddelimiter, wordchar,
columndelimiter, columnchar, linedelimiter, linechar, whitespace, word, line,
and column. This problem still exists in the perl pattern library, and can be
worked around by using \x0b instead of \v.

What's New in TextPipe v7.0.7
=============================

*
Expand Zoned Decimal and Expand Packed Decimal filters are no longer added
inside a Restriction filter.
*
Changed EasyPatterns for EmailAddress, HyperLink, PageNumber and DayOfYear so
that the constituent parts are captured into $variables for later
substitution.
*
Fixed EasyPattern for FormFeed.
*
Fixed handling of EBCDIC negative zoned decimals.

What's New in TextPipe v7.0.6
=============================

*
Fixed bug with parsing of text with \x{...} characters.

What's New in TextPipe v7.0.5
=============================

*
Fixed shell context items - the last item did not work. Also fixed the path
for the Split File filter.
*
Fixed bug with Rolling back .BAK files - if the Only Output Modified Files
option was on, it would delete original files.

What's New in TextPipe v7.0.4
=============================

*
Unicode conversion filter was only outputting the Byte Order Mark for the
first file.

What's New in TextPipe v7.0.3
=============================

*
Added swap button to Unicode conversion filter.
*
Fixed Unicode conversion filter bug.

What's New in TextPipe v7.0.2
=============================

*
Fixed Backup Mode behaviour when the .bak file already existed (now it is
deleted).

What's New in TextPipe v7.0.1
=============================

*
Fixed Hex Dump option for Edit Fields and File Grid.

What's New in TextPipe v7.0
===========================

*
New Unicode (UTF-16LE) search and replace, with options for space compression
and for ignoring non-spacing characters.
*
New Unicode (UTF-16LE) pattern matching.
*
Unicode conversion filter is now much simpler, as you just choose an input
and output format in one step.
*
New XSLT filter for manipulating XML.
*
New file grid right-click menu items:
- Analyze file - checks for various Unicode, XML, CSV and Tab delimited
formats, Unix/PC/Mac end of line format and other file types
- Hex dump - displays a hex dump of the first 200 characters of the file
*
Added \x{...} Unicode character matching for exact matches. This means that
for a Unicode space you can type \x00000020 instead of \x00\x00\x00\x20.
*
Synchronized scrolling option for Trial Input and Output.
*
You can now control if the Trial Input and/or Output are treated as Unicode.
*
Optimized single byte maps for the special case of one output character for
every input character.
*
Better help contexts for the 9 different search/replace types.

What's New in TextPipe v6.9
===========================

*
Removed restriction on pattern recursion depth - this was causing Stack
Overflow errors for legitimate expressions.
*
Fixed file grid ordering - if you moved rows around, TextPipe did not
remember the new row position.
*
Fixed menu items for Remove, Copy and Move tab fields. These did not add the
required filters.
*
Fixed help hyperlinks on Trial Run and Files tabs.
*
Fixed bugs in Unicode Make Little/Big Endian filters.
*
Fixed bug in Convert From Unicode filter, for when it encountered a character
that could not be converted. It now outputs a space.
*
The EasyPattern [year] keyword now checks for a 4 digit year before checking
for a 2 digit year.
*
Convert Word Document To Text filter now only closes the Active Document,
avoiding the MS Word error message 'This method or property is unavailable
because a document window is not active'.
*
Fixed line wrapping problem with the clipboard export text of several
restrict-type filters.
*
Increased maximum stack size to cope with increased size of Unicode filters
(and avoid Stack Overflows).
*
Fixed database filter not quoting char or date/time data fields.

What's New in TextPipe v6.8.3
=============================

*
New filters for handling pipe-delimited files:
  Remove pipe fields
  Restrict to pipe fields
  Copy pipe fields
  Move pipe fields
*
Added EasyPattern keywords for slash, colon, hash, pound, percent, star,
asterisk and ampersand. EasyPatterns now generate much better perl regexps.
*
EasyPatterns can now set the maximum expected match size - this allows them
to match text larger than the default of 4K.
*
Increased processing speed for Trial Runs and Clipboard output.
*
Restrict to fields filters now export the settings for Process Fields
Individually, Exclude Delimiter, Exclude Quotes.
*
Fixed EasyPatterns [month] and [monthname] to attempt to match a long month
name (January, February) first, then a short month name (Jan, Feb).
*
Fixed bug with Context Grep with lines after the match. Matching lines at the
end of the file were being omitted.
*
Fixed code for handling CSV, Tab and Pipe delimited files with header lines.

What's New in TextPipe v6.8.2
=============================

*
Added new EasyPatterns for [ CSVfield ] and [ TABfield ].
*
New Restrict to Byte Range filter - useful for expanding mainframe packed and
zoned decimal fields.
*
New Delete Byte Range filter.
*
XML filter now has better matching of double and single quoted attributes.
These filters now handle embedded double single or double quotes, quotes
escaped with backslash, and embedded new lines.
*
Optimized expand packed decimal and expand zoned decimal for speed.
*
Fixed bug with swap UTF-16 and swap UTF-32 word order.

What's New in TextPipe v6.8.1
=============================

*
Added Expand packed decimal and Expand zoned decimal filters.
*
Fixed handling of Restrict to between tags filter - under some conditions the
trailing > of the start tag would be put in the replace result.
*
Fixed Copy CSV/Tab fields bug. When copied fields were appended, the
delimiter was not inserted if the copy position was the next available field.

What's New in TextPipe v6.8
===========================

*
Added Unicode functions
  - Remove BOM (Byte Order Mark)
  - Swap UTF-16 word order
  - Swap UTF-32 word order
  - Make Little Endian
  - Make Big Endian
*
Added 144 Unicode conversion functions to convert to and from:

ARMSCII-8       ASCII          BIG5              BIG5HKSCS
C99             CES-BIG5       CES-GBK           CNS-11643-1
CNS-11643-15    CNS-11643-2    CNS-11643-3       CNS-11643-4
CNS-11643-5     CNS-11643-6    CNS-11643-7       CNS-11643-INV
CP1046          CP1124         CP1125            CP1129
CP1133          CP1161         CP1162            CP1163
CP1250          CP1251         CP1252            CP1253
CP1254          CP1255         CP1256            CP1257
CP1258          CP437          CP737             CP775
CP850           CP852          CP853             CP855
CP856           CP857          CP858             CP860
CP861           CP862          CP863             CP864
CP865           CP866          CP869             CP874
CP922           CP932          CP949             CP950
DEC-Hanyu       DEC-Kanji      EUC-CN            EUC-jisx0213
EUC-JP          EUC-KR         EUC-TW            GB18030
GB2312          GBK            GEORGIAN-ACADEMY  GEORGIAN-PS
HP-ROMAN8       HZ             ISO-2022-CN       ISO-2022-CN-EXT
ISO-2022-JP     ISO-2022-JP1   ISO-2022-JP2      ISO-2022-JP3
ISO-2022-KR     ISO646-CN      ISO-646-JP        ISO-8859-1
ISO-8859-10     ISO-8859-13    ISO-8859-14       ISO-8859-15
ISO-8859-16     ISO-8859-2     ISO-8859-3        ISO-8859-4
ISO-8859-5      ISO-8859-6     ISO-8859-7        ISO-8859-8
ISO-8859-9      ISO-IR-165     JAVA              JIS_X0201
JIS_X0208       JIS_X0212      JOHAB             Johab-Hangul
KOI8-R          KOI8-RU        KOI8-T            KOI8-U
KSC-5601        MacArabic      MacCentralEurope  MacCroatian
MacCyrillic     MacGreek       MacHebrew         MacIceland
MacRoman        MacRomania     MacThai           MacTurkish
MacUkraine      MULELAO-1      NEXTSTEP          RISCOS1
SHIFT-JISX0213  SJIS           TCVN              TDS565
TIS-620         UCS-2          UCS-2BE           UCS-2-Internal
UCS-2LE         UCS-2-Swapped  UCS-4             UCS-4BE
UCS-4-Internal  UCS-4LE        UCS-4-Swapped     UTF-16
UTF-16BE        UTF-16LE       UTF-32            UTF-32BE
UTF-32LE        UTF-7          UTF-8             VISCII

*
New context menu item to insert control characters like tabs and form feeds.
*
New context menu item to show a hex dump of the selected text.
*
Added logWarning to scripting model.
*
Added Cut/Copy/Paste to file grid context menu.
*
Added Tile function to Window menu.
*
The Convert Word Documents To Text filter now disables Word's screen display
for faster processing.
*
The TextPipe Engine DLL now has a COM interface identical to TextPipe's, but
removes most GUI operations for incredible speed.
*
Added alert() function to the internal TextPipe object of the scripting
filter, so JScript functions can easily display debug dialogs.
*
EasyPatterns bug fixes and improved flexibility. In particular, the [char]
keyword was fixed.
*
Several filters that should have allowed subfilters did not. Filters affected
were: database filter, insert column filter, add left margin filter.
*
Fixed bug - the filename was being output even for non-matching lines, when
context lines were included.
*
Fixed bug with partial trial run menu items including too many items.
*
Fixed bug with the filter tree allowing nodes to be pasted on the Input
Filter, resulting in error-producing subfilters of the Input filter.

What's New in TextPipe v6.7
===========================

*
EasyPatterns - English-like text patterns for mere mortals e.g. to match
phone numbers, use
  [3 digits, punctuation, 3 digits, punctuation, 4 digits]
As easy as that!
*
Many fantastic new HTML and XML filters!
* Restrict a search/replace or other filters to an XML element, attribute or
to text between XML tag pairs
* Remove an XML element, attribute or XML tag pair.
*
New Restrict To Filename filter. This allows you to search and replace inside
filenames, number and rename files and much more!
*
Very powerful new debugging feature - selective trial runs. You can select
which filters are being tested without having to clone a piece of the filter.
You can perform a Trial Run on:
* Entire Filter List (current behaviour, default)
* Selected Filter Only
* Selected Filter and Sub-Filters
* Filter List up to and including Selected Filter
To see these new options, right-click on a filter in the tree view.
*
New Backup Mode - renames original files to .BAK. Easily restore originals
with a single click in the Results window.
*
New Convert Word documents to text files filter.
*
Fixed bug when the Result window Cancel button was clicked twice in a row.
*
Fixed bug with pasting filter tree items that were copied from level 2 or
deeper.
*
Fixed bug with the Windows Explorer Extension that showed the TextPipe menu
multiple times.
*
Fixed bug with multiple consecutive Script filters not passing header text.
*
Fixed bug with Search/Replace Enable UTF-8 option and Extract option being
swapped when saved.
*
Fixed loading of Unicode maps from .TAB and .CSV formats - blank fields were
erroneously skipped.

What's New in TextPipe v6.6.5
=============================

*
Fixed bug with Remove Duplicates option of Sorting filter not being
respected.

What's New in TextPipe v6.6.4
=============================

*
New macros @time, @date and @datetime.
*
Fixed bug with handling of CSV fields with more than one line embedded in a
column.

What's New in TextPipe v6.6.3
=============================

*
Vastly improved sorting performance for files larger than 50 MB.
*
Simplified user interface by removing
1. Comments tab (old comments are now inserted into the filter list as the
first item)
2. Forcing the MDI child window to initially appear maximized, making it look
like a SDI application. The window size can still be changed as before
3. Removed the title that appeared above the properties
4. Removing the scratch pad tab unless a loaded filter has text in the
scratch area.
*
Added UTF-32 Unicode Maps
*
When Prompt On Replace is on, it can now be disabled during the search. Note
- this does not update the tree view immediately, and if the filter is
currently selected it has no effect.
*
The evaluation version provides a warning if the user attempts to modify
Excel or Word documents.
*
Unicode maps can now append multiple map files together into one map.
*
Unicode maps can now log overlapping regions as warnings or errors.
*
Improved Unicode map timings for populating with Default or blank values.
*
Added multi-maps for mapping input Unicode to multiple output Unicode forms
depending on command line parameters.
*
/MINIMIZED now examined before any other parameter, eliminating flicker.
*
Fix error when the file grid was selected in the askSam version.
*
Fix for Run Time version that prevented filter additions after a filter had
been loaded with /F=.

What's New in TextPipe v6.6.2
=============================

*
New File Grid context menu option to copy 100 Kbytes to the Trial Run Area.
*
Upgraded to PCRE 4.3 pattern matching library to fix assertions not working
correctly.
*
Added warning message that not all filter types can be exported to the
command line.
*
Tips and Wizard dialogs now allow filters being double clicked on from
Windows Explorer to be loaded.
*
Tips and Wizard are only shown if a filter hasn't been automatically loaded.
*
Run Time version now disables sound by default and does not show a tray icon.
*
Filter overlay images are now updated correctly when a disabled sub filter is
deleted.

What's New in TextPipe v6.6.1
=============================

*
Added Exit Codes depending on the highest level of error logged.
*
New /EXPORTFILTERTOLOG command line option.
*
Changes to support TextPipe Engine file processing.
*
Trial Input now preserves Unix EOLs when a file is pasted or loaded.
*
Changed menu structure to make it easier for new users. Removed some icons
from the toolbar.
*
Fixed Access Violations with Remove Lines Matching Perl Pattern filter, and
on some random occasions, with Perl pattern matches.
*
Fixed code generation of setPerl() statement.
*
Fixed an extra blank line being inserted at the start of the next file after
a Unicode file was processed.

What's New in TextPipe v6.6
===========================

*
New filter supporting search/replace lists. The lists can either be specified
in a grid, or they can be loaded dynamically from Excel, CSV or Tab-delimited
file formats at run-time.
*
The file list now supports multiple file specifications separated by semi-
colons eg c:\INetPub\*.html;*.txt;*.htm
*
The Split Files filter can now control how the output files are named.
*
Subtract entries in the file list are now shown in red, and Ignore entries
(comments) shown in grey.
*
The Script filter now uses a Syntax highlighting editor to display VBScript,
JScript, VBA, PerlScript and PythonScript code. It also allows separate
default scripts to be saved for each language.
*
Unicode maps can now be loaded from and saved to Excel/CSV/Tab formats.
Unicode filters are now far faster and can deal with undefined characters -
such as removing them, replacing them with a custom string, sending them to a
VBScript sub filter for specialized processing, or allowing them to pass
through unchanged.
*
The filter tree view now supports Cut, Copy and Paste clipboard operations.
There is also an undo action for the current filter. Filters can also be
pasted into any application as a text export.
*
CSV and Tab-delimited files with CR/LFs embedded in fields are now handled
correctly. New restriction filters can be used to remove the embedded CR/LFs.
*
CSV and Tab-delimited restricted filters can now remove the comma or tab
field delimiters and the surrounding quotes to make it easier for sub filters
to work with field data. Each field's data can now be processed individually
rather than as a concatenation of field data.
*
Byte map lines where the output is different to the input are now shown in
yellow. Byte maps can now scroll directly to the character you press on the
keyboard - which makes finding mappings easier.
*
In the audit log, merge file filters now summarize the changes made by
filters that follow them in the filter list. Split file filters also
summarize the changes made.
*
GUI and internal changes to support the new French translation.
*
New command line options /CLEARCOMMENTS, /CLEARFILTERS, /SAVEFILTER,
/LOGENABLE, /LOGMODE, /LOGTHRESHOLD, /PREPAREFILTERFORDISTRIBUTION,
/SAVEFILTER, /PLAYSOUND.
*
When loading a list of commands from a command file with /Z=, comments can now be included in the file using a leading ';'.
*
Speed improvements for the GUI and for filter execution.
*
Audit logging now shows status for more filters including Scripts and Maps.
The audit log text shown for search/replace filters now matches what is shown
on the screen for ease of reference.
*
Various COM scripting changes to support new features, and to allow DataPipe
to set the current directory.
*
Perl patterns now support UTF-8.
*
Convert end-of-lines filter now handles Mainframe CMS V- and F-format files
(downloaded to PC in EBCDIC format).
*
Removed Rich Edit Memo controls - they caused RTF codes in plain text to be
interpreted as fonts, and incorrectly added extra CR/LFs to Add Margins-type
filters.
*
The file list had a bug where it added file entries more than once if they
matched multiple file specifications in the same group.
*
Fixed bug with loading of single byte maps where the map was not actually
loaded, from both the menu and from the map sub-form. Fixed Reverse function.
*
Unicode Maps - fixed problem where changes to the grid were not detected and
saved.
*
Fixed bug with importing Excel and Tab-delimited search/replace lists.
*
Fixed bug with appending filters from the command line with multiple /F=
appearing in reverse order. Fixed filters such as /Hx not being added.

What's New in TextPipe v6.5.1
=============================

*
New filters - CSV to Tab-delimited, CSV to XML, Tab-delimited to CSV, Tab-
delimited to XML.
*
Fixed error message when egrep pattern matcher encountered a \000 (null).
Only perl patterns can handle nulls.

What's New in TextPipe v6.5
===========================

*
Search and replace lists can be loaded easily using the Load List from
Excel/Tab/CSV formats in the Replace menu.
*
The Task Bar Menu and Map Menu now display sub folders of the TaskMenu and
Maps folders, and sort all the filenames and folder names shown. This makes
it very easy to categorize these maps and filters into folders.
*
If a filter has one or more of its sub filters disabled, a red slash is shown
through its icon in the tree view. This makes it easy to find filters where
disabled sub filters are hidden from view. A disabled filter still has a red
cross through its icon in the tree view.
*
Custom patterns can now be defined in the file custom_patterns.txt in the
TextPipe.exe folder. The patterns can be added through the right-click menus
on every field.
*
Sorting can now handle files of unlimited size. Previously sorting was
restricted to using physical and virtual memory.
*
New sorting options for date and time, date, and time.
*
New right-click option for filter tree view to turn a filter into a sub
filter.
*
Matching/non-matching lines and other grep line based filters can now use
Perl patterns, egrep patterns, MS Word patterns and Brief patterns. The
addGrepFilter COM function has changed to support this.
*
Export to VBScript/JScript/web page now also exports the search replace
options such as greedy matching, allow comments, buffer size etc. The edit
distance threshold parameter has been removed from the addReplaceFilter COM
function, and new functions added for setPerl, setEditDistance and
setBufferSize.
*
Menus and tabs re-organisation, selection filters renamed to restriction
filters.
*
A fixed width font is now used for most memo fields.
*
Improved handling of malformed CSV files.
*
Fixed export of Logging:Append to Log. The wrong value was being exported.
*
Fixed generation of random files of specified size. All files were being
truncated at 8k.

What's New in TextPipe v6.4.1
=============================

*
Added "Content Type" to system registry to help web downloads of .fll files.
*
COM object now registers itself in the Running Object Table (ROT), and frees
itself at the correct times.
*
New COM properties Application.visible and Application.activeFilter
*
Fixed bug with moving nodes with sub filters.
*
The Task Bar menu now correctly runs filters for the second and subsequent
runs - previously the status window stopped it from working.

What's New in TextPipe v6.4
===========================

*
New filter to Remove a line range (in addition to the Head/Tail filter).
*
New filter to Remove CSV fields.
*
New filter to Remove Tab fields.
*
New filter to Move CSV fields.
*
New filter to Move Tab fields.
*
New filter to Move Columns.
*
New filter to Copy CSV fields.
*
New filter to Copy Tab fields.
*
New filter to Copy Columns.
*
File/Open command can now open multiple filters at a time.
*
Drastically improved the speed of 'head' filters and Remove All - they now
skip reading the remainder of the file no matter how large it is (a 'Head'
filter is used to extract the first N lines of a file).
*
New option to display help in an embedded window that is synchronized with
the current action.
*
Added filter export to VBScript and JScript - this can be used to very easily
create code for ASP or web pages or simple scripts.
*
Added COM interface to create Head/Tail filter, Run External Program filter,
Split filter, Show/Remove Duplicates filter, Secondary Output filter. Logging
functions, Input filter and Output filter settings can now be controlled with
precision.
*
COM functions are now error trapped to prevent dialog boxes appearing in
server versions. Error messages are captured to the ErrorText property.
*
The Script Filter's default script can now be set.
*
Fixed display issue where memo field containing multi-line text (such as the
search and replace fields) scrolled to the bottom rather than displaying text
from the top.
*
The command line text of the External Program filter now does not have double
quotes removed from it.
*
TextPipe now can paste more than 64K of text at a time into a memo control.
Previously controls could hold more than 64K of text, you just couldn't paste
that amount.
*
TextPipe now forces text paste operations to be plain text.
*
Speed improvements with string translation caching.
*
Wizard form now aligns correctly.

What's New in TextPipe v6.3.6
=============================

*
A new option to only output modified files was added to the Output filter.
*
A new Search option on the File Menu allows filters to be easily located.
*
The status window now shows how many bytes of the input file have been
processed.
*
A quick short cut is now to double click on any of the filename fields to
simply open the file (e.g. Single file output).
*
The Remove and Remove All buttons now work for all tabs (previously they were
not used for the Comments, Scratch Pad and Trial Run tabs).
*
When filters are being run from the command line, a new blank filter window
is no longer created - this means that the status of the already running
filter can be easily seen.
*
The Execute Program filter now deletes the Input and Output files when
processing has completed. This filter is commonly used for sorting very large
data files and other external processing.
*
The context grep filters Remove matching/non-matching lines and Retain
matching/non-matching lines have been re-designed so that they work exactly
the same as the Select matching/non-matching lines filters.
*
The /I command line option (recurse subfolders) now also applies to file
lists loaded using /L=.
*
Added new command line option /CLEARFILES to clear the file list. Also added
/S0-/S6 to handle different search/replace types.
*
Fixed bug with Remove/Retain Lines Matching Perl Pattern List, where all
backslashes needed to be doubled twice.
*
Fixed bug with none of the /C... command line parameters working.

What's New in TextPipe v6.3.5
=============================

*
Fixed problem with Extract Email Addresses filter removing one extra
character from the start of the email address when it started with mailto:
sendto:, email= etc.
*
Improved window behaviour when TextPipe is activated from COM - the main
window is now not visible unless it is specifically shown.
*
Improved COM handling - TextPipe exits correctly now that there are no
internal references to the Application object.

What's New in TextPipe v6.3.4
=============================

*
Fixed problem with saving Count Duplicate Lines and Remove Duplicate Lines
filters.

What's New in TextPipe v6.3.3
=============================

*
Fixed problem with Remove Patterns From List filter - was crashing.

What's New in TextPipe v6.3.2
=============================

*
Fixed problem with multiple output filters overwriting the same file.

What's New in TextPipe v6.3.1
=============================

*
Fixed sub filters not working for filters Add Text Side by Side, Add
Repeating Text Side by Side, Remove Lines Found In, Retain Lines Found In,
Remove Matching Lines Found In, Retain Matching Lines Found In.
*
Added new options to the Export Command Line of the Output filter.
*
Fixed problem with memos not displaying text properly.

What's New in TextPipe v6.3
===========================

*
Added new Find in HTML helper window to help users Edit, Remove, Replace or
Append to text inside HTML tags (such as the ALT tag of IMAGES), between HTML
tags (such as between  and ), and between two arbitrary pieces
of text.
*
Added new 'Select each line in turn' filter. This is a short cut for creating
a 'Select matching lines' filter with a pattern of '^'.
*
Window Menu now works properly - workaround found for Windows bug (being to
turn off AutoHotKeys and AutoLineReduction).
*
New System menu item (on task bar) to force TextPipe to close, even if it has
been activated via COM or from DataPipe etc.
*
Fixed double quotes appearing in filename fields after a filename has been
chosen.
*
When special characters are pasted, the default mode has been changed from
decimal to hex (so where \018 would previously have been pasted for special
character 18, now \x12 would be pasted)
*
New Word Wrap toggle item in the popup menu of memo fields.
*
New Start Column and Length options for Sort filter, Remove Duplicates filter
and Count Duplicates filter.
*
When a command line filter has been started with /G, TextPipe no longer
clears the filter list so that you can now see the contents of the command
line filter.
*
New command line parameter /OUTPUTKEEPSTRUCT to control whether the input
file's folder structure is retained when a new output folder is set.
*
Fixed a bug with dragging and dropping filters, where you could get an Access
Violation trying to edit the filter after moving it from inside its parent
filter.
*
Fixed @inputFilename and @fullInputFilename incorrectly using the output
filename.

What's New in TextPipe v6.2.9
=============================

*
Expanded the Special\Find and Replace menu item into an entire menu to make
it clearer.
*
Now uses Windows standard dialog boxes for easier translation.
*
Removed status window control resizing.
*
Removed validation on perl patterns that prevented back references from
working.
*
Fixed error with perl ^ not matching the start of the first line in files
less than 4K.
*
Fixed error with Extract Filter - some non-matching text was being passed to
the output.
*
Fixed the maximum allowed value on various filters - Split At Lines, Detail
lines allowed before writing to log file, size of random input file that can
be generated, insert column position, line numbering, context grep lines
before and after, delete columns values, duplicates filter, record length on
EOL filter, head/tail filter, all number-type filters like spaces to tabs,
centering etc, maximum buffer size for pattern matching etc.
*
Fixed handling of single and double quotes inside database queries.
*
Database filter 'insert script' option was not modifying file changed status.
*
Fixed TextPipe.getGlobalVar - now returns a blank string if the variable has
not been defined, instead of the variable name preceded by '@'.
GlobalVariables are now case-insensitive, and can hold multi-line values.
*
Fixed bug with 'Remove Prompting' not setting the modified flag if the filter
changed.
*
Changed the text qualifier for database insert scripts from " to '

What's New in TextPipe v6.2.8
=============================

*
All fonts are now set to the system default font. This makes translation of
Eastern languages far easier.
*
Tools\Put Sample In Trial Input now only includes files that have the 'Add'
action enabled. It also handles the situation where no files have been
specified.
*
Better DataPipe integration - now no longer hides the window when TextPipe is
closed. The main form minimizes instead.

What's New in TextPipe v6.2.7
=============================

*
Improved Command Line Wizard dialog.
*
New right click menu function for search/replace filters "Allow RTF tags"
converts the pattern to one that allows embedded RTF font tags.
*
Better integration with DataPipe - COM functions have changed, now
Application.hide minimizes TextPipe, and Application.show restores it.

What's New in TextPipe v6.2.6
=============================

*
Improved speed of COM functions.
*
Added COM code to support DataPipe
*
Fixed bug (again) where null lines caused the pattern matcher to fail.

What's New in TextPipe v6.2.5
=============================

*
Race condition fixed where a command line processing thread started with /G
/Q doesn't get a chance to start before the application exits.
*
New command line parameters /MAXIMIZED and /HIDDEN (hidden acts the same as
/MINIMIZED)
*
COM functions execute() and executeClipboard() now don't return until the
processing thread has completed.
*
New COM function isRunning to determine if a processing thread is still
running.
*
The COM function addReplaceFilter()'s ignoreCase parameter has been renamed
to matchCase - now the sense of the value is correct, and is consistent with
the user interface.
*
Fixed bug with perl-style patterns when used with look-behind assertions -
100 characters of text preceding the match is now retained in the search
buffer.

What's New in TextPipe v6.2.4
=============================

*
Secondary Output Filters can now be disabled and deleted.
*
Fixed errors occurring when a filter was modified after it had been moved up
or down using the up or down buttons.
*
Added Application.Quit function to COM
*
When created from COM, the default is now for the main form to be invisible.

What's New in TextPipe v6.2.3
=============================

*
Secondary Output Filters can now be added throughout the filter list to
duplicate the output file, or write selected lines and columns out to a
separate file, eg to place error lines or records in an error file, or to
separate records of different type or length into new files.
*
A new Select Block filter allows undelimited blocks of data to be selected
and processed as one.
*
Fixed VB/Jscript timeout - value is in milliseconds, not seconds.
*
Fixed obscure Perl replacement bug when run from command line.
*
Validation of ActiveX scripts now only occurs for VBScript - JScript
validation didn't work because the MS JScript engine does not implement
various functions.

What's New in TextPipe v6.2.2
=============================

*
Updated perl library to PCRE 3.7 (from 3.5)
*
EOL filter now allows more than one character in the replacement string.
*
Enabled some command line operations to provide Windows Explorer support in
TextPipe Web and Standard.
*
Fixed problem with the About dialog (in the evaluation version) interfering
with filter loading when TextPipe was invoked by double-clicking or opening a
.fll file.
*
Fixed bug with 'Escape Special Characters' not working properly.
*
Fixed bug with Split After XX Lines filter only outputting one line.
*
Fixed bug with 'Include Lines from End of File' when the number of lines was
greater than 255.

What's New in TextPipe v6.2.1
=============================

*
Tree view now displays more than just the first line of scripts and comments.
*
Fixed bug with perl-style grep and blank lines.
*
Input Filter export showed incorrect value for Delete Input Files After
Processing.
*
Filters loaded from the command line now have their full path saved, so they
get saved to the correct location even if the current directory is changed.
*
Added missing properties to Filter Export function.
*
Fixed bug handling command line filenames with embedded '..' paths.

What's New in TextPipe v6.2
===========================

Note: v6.2 was released quickly to correct some serious bugs, but it was not announced
until v6.2.1 was released with further bug fixes.

*
New filter to Remove lines matching any patterns in a list.
*
New filter to Retain lines matching any patterns in a list.
*
New filters to Retain matching lines, Remove matching lines, Retain non-
matching lines and Remove non-matching lines - this is easier to understand
than the Select lines filters.
*
New filter to add a file side-by-side with an existing file.
*
New filter to add a file side-by-side with an existing file and repeat the
added file if the file being added to has more lines.
*
New filter to Split a File after a given number of lines.
*
New Input Filter to generate random a data file of a particular size.
*
New filter to repeat a file's contents a specified number of times.
*
Added Script Timeout option to VBScript/JScript filter
*
Show Command Line dialog now also shows the current path and all environment
variables.
*
Added 'Escape special characters' menu item to right-click menus of all
fields. This is very handy for adding backslashes in front of pattern
matching meta characters, and doubling '@' and '$' characters where
appropriate.
*
Added 'Allow Extra Spaces' menu item to right-click menu of all pattern
fields, allowing you to easily search and replace in HTML files by ignoring
any extra spaces or tabs that commonly occur.
*
Added 'Allow Extra Line Feeds or Spaces' menu item to right-click menu of all
pattern fields, allowing you to easily search and replace in HTML files by
ignoring any extra spaces, tabs or line feeds that commonly occur.
*
Updated help, fixed application links to help file, help file now displays
when TextPipe is first run
*
Fixed bug with filter loading
*
Made splitter in Search/Replace filter visible, now resizes correctly.
*
Fixed 'Add folder' button - newer versions of Windows caused this dialog to
appear behind the main window.
*
Unknown field codes (a name preceded by an '@' symbol) are now NOT replaced
with a blank - they are left as is to prevent inadvertent use of a field code
causing text to disappear. Known field codes (@clipboard, @inputFilename,
@fullInputFilename) still work as normal.
*
Added new COM function to filter windows - closeWithoutSave. Added new
options to COM interface, including default values.
*
Fixed bug with a filter validation function not being called when there were
sub filters anywhere above them in the filter list.
*
Fixed a bug affecting the Select Matching Lines, Select Non-matching Lines,
Select Lines From Length To END, Select Lines From Start To End, Select Lines
From Start For Length filters when immediately followed by a line-based
filter such as a Reverse Each Line, Pad To Width etc. Some text was not
output at all.
*
Fixed a bug affecting the Remove Lines Found In List and Only Keep Lines
Found In List filters - because the list was not internally sorted, the
actual lines would only occasionally be processed correctly. A workaround was
to ensure that the line list is first sorted in ascending alphabetical order.

What's New in TextPipe v6.1.1
=============================

*
Improved the email address extraction filter to automatically remove leading
'mailto:', 'mime:', 'smtp:', 'email=', and remove trailing '?Subject' and
trailing periods.
*
Fixed Perl Pattern Match not saving 'greedy' setting
*
Fixed hidden form problem with active field being invisible
*
Fixed Access Violations on editing filter list

What's New in TextPipe v6.1
===========================

*
PERL-STYLE pattern matching! - use both greedy and non-greedy matching
options, as well as embedded comments, assertions, non-capturing sub-
expressions and more!
*
Brief-style pattern matching
*
MS Word style pattern matching - use the *, ?, etc wildcards that you can use
in MS Word.
*
New 'Extract' option for search and replaces, all non-matching text is
discarded.
*
The Found Text can now be referred to in the Replacement Text as '$0' for
Exact, Sounds-like and Edit-distance matches. Previously the Found Text could
not be referred to, making some replacements harder.
*
You can now easily add a sub filter to the selected filter by holding [Alt]
down while choosing it from the menu or selecting its speed button.
*
You can now easily add a filter at the parent level of a sub filter by
holding [Shift] down while choosing it from the menu or selecting its speed
button.
*
New Input Filter setting for text to come from the Trial Run Input area.
*
The filters Add Left Margin, Add Right Margin, Add Header and Add Footer now
do not parse the included file for special functions or characters. This
ensures that the file being included is included exactly as is. The parsing
of text specified in the edit field continues as previously.
*
Fixed a recursion bug when [Alt+Down] is pressed to move a filter and its sub
filters down, now the filters are moved down correctly.
*
Fixed bug with base panel of search and replace filter disappearing after
returning from the Trial Run area.
*
Removed the Font Control context menu item for controls.
*
Better handling of Cancel button during a processing run.
*
Fixed handling of NULL values with database filter and Generate Insert Script
mode.
*
Database Command Timeout setting for queries that run for more than 30
seconds.
*
Improved Tree handling is quicker and easier to use; the treeview changes
immediately when a setting is changed, and there is a new Up and Down button
to easily move filters up and down (in addition to Alt+Up and Alt+down).
*
All command line defaults now come from the Normal Template normal.fll, not
the internal defaults which used to be:
  inputFromFiles, promptOnEach=false, skip binary files,
binarySampleSize=100, outputToFiles, testMode=false, retainDateTime=false,
changeExtension=false, changeBaseDir=false.
*
Fix for the error " The Decision Cube Capacity is Low. Please deactivate
dimensions or change the data set"

What's New in TextPipe v5.6.3
=============================

*
Uninstall now removes all registry entries
*
Installer now supports unattended installs
*
Fixed short cut conflict between Remove/Remove All buttons and menu items
*
Fixed bug with registry security in registered version

What's New in TextPipe v5.6.2
=============================

*
Added 'Normal Template' functionality - a filter that is always loaded to set
defaults.
*
Delete Input Files after Processing setting now displays a warning box when
it is enabled. When a new filter is created this setting always defaults to
disabled.
*
Added shortcuts to tabs on status form, query replace form and command line
display form.
*
Fixed error in COM documentation for AddFile function - constants should be
1,2 and 3, not 0, 1 and 2.
*
Changed sizes of GUI controls to show foreign languages better.
*
Fixed bug with loading of foreign language, and enabled support for radio
group type controls.
*
Fixed bug with Append Mode.

What's New in TextPipe v5.6.1
=============================

*
Clicking on the filename column in the file list now sorts the filenames.
*
Improved COM interface - new methods for compiling the filter list which
improves speed if the same filter is used repeatedly
*
New Append to existing file setting

What's New in TextPipe v5.6
===========================

*
Localisation complete - now everything in TextPipe can be translated to a
foreign language by translating the file Languages\English.ini

For more details on how to translate TextPipe, please visit:
  http://www.datamystic.com/textpipe/translate.html
*
COM interface now allows TextPipe to be fully controlled from languages such
as VisualBasic, VBScript, C++, Delphi etc

For more details on how to translate TextPipe, please visit:
  http://www.datamystic.com/textpipe/com_interface.html
*
Fixed bug with dragging and dropping filters onto a filter that doesn't
support subfilters - now adds the dragged filter above it

What's New in TextPipe v5.5.2
=============================

*
Added code for TextPipe Pro Run Time Engine
*
Marketing changes for evaluation edition
*
Began preparation for localisation
*
Fixed command line processing bug with modifying the Output filter settings
after loading a filter with the /F= command
*
Fixed bug with debug windows

What's New in TextPipe v5.5.1
=============================

*
Forms and menus can now be translated into other languages via a .ini file in
the Languages folder
*
Improved the load time of massive filter lists loaded via the /Z= command
line parameter
*
Trial Output now shows Column and Row in the same way that Trial Input does
*
File size for files larger than 2GB is now displayed correctly
*
Improved the way that form sizes and positions are saved
*
Improved the way that registered user data is saved under Win2K
*
Forced installation to require the Administrator account to prevent registry
writing problems under Win2K

What's New in TextPipe v5.5
===========================

*
Lines of context surrounding a matching line can now be displayed, both
before and after matching or non matching lines (grep filter)
*
Added short cuts to tabs (Alt+1 through Alt+5)
*
Added Filter Wizard to help new users
*
Added command line options for setting the output file extension, output
folder, and merge output filename.
*
Sub filters can now be created by holding down Shift as they are drag and
dropped to a location inside another filter
*
The database filter now directly supports creating a database insert script
from extracted data
*
Improved tutorial lessons
*
TextPipe tutorial now listed in the start menu
*
Fixed bug with default folder not being TextPipe's folder (default was My
Documents folder)
*
Fixed bug with access violation on exit after a filter is invoked from the
command line
*
Fixed bug with Count Duplicates filter missing some lines
*
Fixed bug with Extract Emails filter missing the last email address if it
ended at the end of the file
*
Fixed bug with errors not being reported during trial run - now if errors are
reported the status dialog will not automatically close
*
Changed default extension to .fll for link filter dialog. Also added an open
button so that the filter can be easily loaded and edited

What's New in TextPipe v5.4.1
=============================

*
Can now CHANGE THE REPLACEMENT TEXT during a run. This is very useful for
fixing up mistakes in the replacement text, and can also be used for pattern
matches
*
New Paste Special command on context menus to allow pasting of special
characters without TextPipe converting them to its internal code
*
Unknown '\' entries are now output with the leading '\' rather than without.
This is to help new users
*
Trial Input and Output no longer attempt to validate special characters such
as \t etc.
*
Fixed bug with '\' being incorrectly interpreted in the Prompt for replace
window
*
Fixed bug with filters becoming modified when Search/Replace filter was
selected (not edited)

What's New in TextPipe v5.4
===========================

*
VBScript/JScript code can now control TextPipe - LogError, LogInfo,
inputFilename, fullInputFilename, terminate, setGlobalVar, getGlobalVar,
subFilterEntireText, version
*
VBScript/JScript code can now have subfilters. Subfilters can be turned on or
off by the script at any time using InitSubFilters, FlushSubFilters,
subFilterChunk
*
Find/Find Again in all memo fields (multi-line edit)
*
Font control dialog on all edit fields (including memos)
*
Extended the number of filters than can be invoked via the command line
*
Facility to generate TextPipe command line parameters from an existing filter
list
*
Trial Run can now be interrupted - runs in separate thread
*
Ctrl+A (Select All) now works in all fields
*
More options in the Window menu
*
Adhoc Transform tab renamed to Trial Run tab
*
Task bar menu now sends TextPipe to the back after a menu item is selected
*
Better handling of maximized windows when TextPipe restarts
*
Fixed bug with Select lines filter - if the input had (say) 2 lines and the
setting was from line 3 to END

What's New in TextPipe v5.3.9 Beta
==================================

*
Added drag and drop between filter windows
*
Added drag and drop promotion/demotion and movement of filters in the filter
tree display (and removed the Up/Down buttons)
*
Added Lesson 4 to tutorial

What's New in TextPipe v5.3.8
=============================

*
"x2" function now duplicates sub filters at the same time
*
Fixed bug with Insert Column At filter where the column was set to zero (ie
insert at end of line) and the actual line length = 1 (the text was inserted
at the beginning of the line)
*
Fixed bug with replace filter and whole words option - output text can be
duplicated
*
Fixed bug with merge filter and single file output filter
*
Fixed bug with Close All function not closing all windows
*
Fixed bug with Insert Filter File function changing the name of the currently
loaded filter

What's New in TextPipe v5.3.7
=============================

*
Updated Export To Clipboard and Print facility to cope with new filters and
tree structured filters. This facility is an excellent way to maintain
version information about your filters
*
Maps now moved to separate 'maps' folder
*
textpipe.log is now created in the same folder as textpipe.exe if no path has
been specified
*
Recent Files menu now uses submenu of main menu
*
Filters opened will now load into the New Filter window if it is unchanged
and does not hold a filter
*
Fixed problem with files and folders added via the command line not
defaulting the subfolders setting correctly
*
Fixed problem with filter opening twice when opened from Explorer when
TextPipe isn't currently opened
*
Fixed bug with merge filter not performing any filters following it in the
filter list

What's New in TextPipe v5.3.6
=============================

*
Added ability for Windows Explorer to detect TextPipe and open a filter in an
existing TextPipe window
*
Recycle bin functionality temporarily removed while bugs fixed

What's New in TextPipe v5.3.5
=============================

*
Fixed buttons and help for Files Tab
*
Delete file operations now use the Recycle Bin
*
Fixed bug with Remove Lines From File and Only Show Lines From file filters

What's New in TextPipe v5.3.4
=============================

*
(Hopefully) fixed long-standing error on exit

What's New in TextPipe v5.3.3
=============================

*
New subscribe facility on help menu
*
Fixed regexp bug with characters sets [] involving ranges of characters
specified with a \ eg [\000-\255]. Also error message for character sets
containing a \000.

What's New in TextPipe v5.3.2
=============================

*
Better error handling when file delete/rename fails
*
Fixed bug with number of subfolders to process (1 Level was same as None)
*
File\Open now always opens a filter in a new window. New menu item to open in
the current window
*
Fixed bug with Remove button, only removed first file from file list even if
multiple files were selected
*
Now checks if log file exists before using Notepad to open it
*
Double quoting of filenames and folders containing spaces now handled
correctly
*
Better handling of dropped files

What's New in TextPipe v5.3.1
=============================

*
Added option to delete input files once they are processed - to support
integration with GFI FAXmaker

What's New in TextPipe v5.3
===========================

*
New Sorting options
* ASCII Sort (case-sensitive and case-insensitive). Previously only an ANSI
sort was available (where lower case letters sort before upper case letters)
* Numeric sort
* Sort by length of line
*
New Unicode Map filter for translating Unicode files
*
Improved online help with Quick Start guide, improved tutorial and more
cross-referencing
*
Status form now has indicator to show whether test mode or overwrite mode is
active
*
All memo fields now can save to and load from a file
*
Trial output now has warning message if the output text contains null
characters
*
Improved Command Line Assistant dialog
*
Fixed bug with Query Replace window not scrolling found text into view
*
Fixed bug with "First Match Only" option discarding text
*
Fixed move filter up/down bug with filter lists containing more than 256
filters
*
Fixed bug with database filter not extracting data correctly when fixed width
format with no column titles was performed
*
Database filter now correctly handles non-SELECT queries that return a record
set
*
Fixed bug with access violation on exit
*
Fixed bug with pattern match ignoring First Match Only setting under some
circumstances

What's New in TextPipe v5.2
===========================

*
Added help toolbar for easier access to helpful features
*
Improved drag & drop behaviour for .fll files
*
Added automation check to disable debug filters
*
Added better error handling for MIME decode quoted printable filter
*
Advanced logging details in results window
*
Added error checking to ensure file can be deleted/moved to correct final
filename, or error logged if not
*
Added output option for all files merged to a single file
*
New Clear All and Paste context menu item
*
Prompting dialog for Read Only files, Prompt Before Processing and Prompt on
Binary Files now has option for No To All and Yes To All
*
New Search/Replace option to only replace the first occurrence
*
Trial Input/Output now have horizontal scroll bars
*
Grep filters (Keep matching lines, remove matching lines) now as select
filters, where matching lines get subfilters applied to them
*
ActiveX scripting no longer requires dummy parameters for functions
*
Improved match information tab for Edit Distance and Sounds-like matches
*
Context menu on tree view for faster deletion/enable toggle
*
Database query now creates correct XML document. Handling of previously
unrecognised data types. Original query output as comment. Headers always
output even if 0 rows returned. Unique column names generated for all output
types. Encoding="UTF-8" removed from XML header - not supported by some
versions of IE
*
Status dialog now shows count of Non-binary Files Skipped
*
New Edit Distance option for search/replace filters - very handy for finding
and locating misspelt words
*
New option to retain the existing file list when opening a new filter
*
New facility for search/replace to replace all or none within the current
selection
*
Loading a .fll filter file now opens in the same window
*
Better detection of filter list changed
*
Better handling of loading very old filter lists
*
Added force lines to width filter
*
Added advanced logging facilities with detailed tracking of all changes made
*
Fixed exception with exact match filter
*
Fixed handling of non-select statements in Database filter, and improved
error checking code
*
Fixed directory where preliminary output file created
*
Fixed split filter - was not performing "start file" processing for following
filters
*
Fixed Remove Email Headers - was crashing
*
Fixed bug that affects all line-processing filters - when using Mac files the
first character on each line may be repeated
*
Fixed bug with exact match filter that caused text to be output multiple
times
*
Fixed Win NT bug with popup context menus on memo fields
*
Fixed delete columns filter
*
Fixed handling of grep/inverse grep filters
*
Comment filters now have vertical scroll bars for comment text
*
Fixed subfilter bugs where filter list was not built correctly, missing
several filters
*
Fixed bug with Select Lines filter not outputting its results
*
Fixed bug with Select CSV Fields filter
*
Fixed errors with start and end word for Edit Distance and Sounds-like
filters
*
Fixed split filter
*
Fixed handling of Select Columns with the 4th option
*
Fixed debug filter error
*
Fixed Select CSV/Tab field missing last field
*
Fixed handling of map menu and task bar menu

What's New in TextPipe v5.1
===========================

*
** Sub filters - drag and drop a filter onto another in the tree view and the
dropped filter will then process JUST the output from the parent node. Eg
drop a "Title Case" filter onto a "Search/Replace" filter to change the
search result to Title Case
*
** New Database Connection filter. Attach to any ADO/ODBC database and
execute any SQL command text against it. The result can be output as CSV,
Tab-delimited or XML (with automatically generated DTD schema). All
characters are appropriately escaped, including quotes within strings for
CSV, and XML special characters.
*
Pattern matching now extended to handle up to 36 ($0-9,a-z) bracketed
expressions. This is 26 more than any other product on the market.
*
New Select CSV Fields filter - select a group of consecutive CSV fields and
apply sub filters to the result by dragging and dropping them onto this
filter. Handles commas embedded in quotes etc
*
New Select Tab-delimited Fields filter - select a group of consecutive tab-
delimited fields and apply sub filters to the result by dragging and dropping
them onto this filter
*
New Select Line Range filter - select a group of consecutive lines and apply
sub filters to the result by dragging and dropping them onto this filter
*
New Select Column Range filter - select a group of consecutive columns and
apply sub filters to the result by dragging and dropping them onto this
filter
*
New Remove All filter - useful for subfilters
*
Convert EOL filter can now specify any string as the new EOL character
*
New Truncate to Width filter
*
File output has new "Maintain folder structure" option when outputting to a
new folder
*
Remove Columns filter now has more advanced options
*
New option Skip Prompt if Identical for Search/replace filters
*
ActiveX scripts can now easily output text at the start and end of each file
*
New tools added - option to verify a filter before it is run - performs
syntax check etc. Also added Windows calculator
*
Explorer extension - right click on files, folders or drives brings up
TextPipe options. New window on Tools menu to customise the Explorer
extensions
*
Command Line Assistant window - provides examples of how to run TextPipe from
the command line in a variety of modes
*
Show Command Line window - displays the command line parameters used to
invoke TextPipe - useful for debugging command line scripts
*
Check for TextPipe Updates window - check if you have the most recent
TextPipe by querying the DataMystic web site
*
TextPipe now interfaces to 2 external tools (see Help\Related Products menu)-
  JustData from www.justsoft.com.au
  Offline Explorer from www.metaproducts.com

GUI enhancements
*
Tree view for filters which allows multi-level filters to be collapsed
*
Multiple-document interface (MDI) allows multiple filters to be open at once.
New Window menu to support MDI interface
*
Dockable toolbars
*
Icons in menus
*
Right-click (context) menus on all edit/memo fields allow special characters,
common patterns, replace expressions and macros to be inserted, as well as
the normal clipboard operations
*
Can now set the font used in the Ad-hoc Input/Output area

Bug fixes
*
ActiveX script now works in non-ad-hoc mode
*
Trial Run area now does not interpret \n, \t etc
*
Split files at Character - fixed bug

What's New in TextPipe v4.6
===========================
*
Example code for the TextPipe Engine - a text transformation .dll that can be
downloaded from www.datamystic.com
*
New ActiveX scripting filter supports arbitrary user-written code in
VBScript, Jscript or PerlScript, PythonScript, REXXscript etc
*
Output To New Folder can now recreate multi-level folder trees and output to
a new drive
*
New Undo command to undo any changes to the current filter
*
New file grid button to Copy Row
*
New Extract filter to extract just the required text
*
Show Duplicates filter now called Count Duplicates, and outputs the number of
times found to the left of each item
*
New Tools menu item to pull sample data from the file list or clipboard into
the trial run area
*
New Tools menu item to move the Trial Run Output back to the Trial Run Input
*
Add Line Numbers filter now allows format of line numbers can be changed to
custom user-defined settings, including numbering in hexadecimal etc
*
Sort filter now has Ignore Case option
*
Header, footer, left margin or right margin can now insert either a file or
text. Cursor position now shown
*
Length of trial input and output area shown in both lines and characters.
Cursor position now shown
*
Warning indicator on status bar now shows if Test Run mode is active or not
*
Debug filter - creates a debug window showing the text passing through the
filter at that point
*
External filter files can now be linked to, providing true 'black box' style
re-use, using File\Link To Filter
*
Can now duplicate the current filter - this makes adding a large number of
similar filters very easy
*
Find and replace filters now use multi-line edit control - making it easier
to edit large selections. The new edit control also provides the length and
line count of the text
*
Maps now display a hex column in addition to the original Decimal and ASCII
columns
*
Changed handling of changing the replacement text for a pattern-based search
- now the default text is not modified for future replaces
*
Tips of the Day
*
Query replace window now supports resizing better
*
Changed registration URLs
*
The filter names shown in the filter list are now truncated, preventing
Windows crashes with very long filter names
*
Fixed logging functions which were going wild when more than 1000 files were
processed
*
Fixed pattern matching not handling \n \r \e \t \a
*
Fixed Randomize filter - was outputting blank text
*
Fixed pattern based search with patterns longer than 247 characters - was
truncating and matching early

What's New in TextPipe v4.5
===========================
*
Soundex search method to search for words that sound like another word.
Useful when you are replacing a typing error
*
New Preferences dialog
- Play sound on completion
- If no association, use user defined editor (need new options window)
*
Prepare filter for automation - removes all user prompting
*
Prepare filter for distribution - removes all test text
*
Can now insert a file with the Add Header, Add Footer, Add Left Margin and
Add Right Margin filters - browse button
*
Add Line Numbers now has option to skip blank lines, (ie omit count but keep
counting), as well as omit count and not count line)
*
Now can split files at a given character, after a specified number of
occurrences (and split before, after or on top of character)
*
ANSI to Unicode and Unicode to ANSI filters (single byte to double byte)
*
Comment filters now shown in italics, disabled filters shown with strike
through text, speed button to add comments
*
Better progress indicator - now much more accurate
*
Duration of filter job in seconds shown in status dialog
*
Increased size of edit fields for easier editing of multi-line text. Display
of text length and number of lines of all memo fields (like Header, Footer,
Left/Right Margin, Insert Column etc)
*
Performance improvement when handling very large files
*
Improved performance for Remove Blanks from Start of Line, Convert to Lower
Case, Convert to Upper Case, OEM to ANSI, ANSI to OEM
*
Query replace dialog now shows file currently being processed
*
Query replace dialog now shows filter information for both normal and pattern
based searches (so it is easy to see which text was matched by each
expression)
*
Improved command line automation support. Now can have comments embedded -
useful when converting Perl scripts to TextPipe scripts because code can be
commented out rather than edited out
*
Form resizing now handled better
*
Fixed bug with Query Replace when replacement text was modified before
replacing
*
Fixed bug with saving/loading of window positions
*
Fixed bug with detection of when the filter has been modified
*
Fixed bug with query replace 'Yes - file' button - did not change the first
entry

What's New in TextPipe v4.4
===========================
*
Name change from DropConvert to TextPipe
*
Regular expression search and replace - allows pattern matching (also known
as non-exact matches or regexps). Case-insensitive or case-sensitive matching
*
HTML Help integration
*
Prompt on replace. Can change the replacement text
*
Option to delete test files at end of test run
*
@inputFilename macro now allowed in filters for Add Text To Start Of Lines,
Add Text At End Of Lines, Add Column Of Text
*
New @clipboard macro to insert the clipboard contents
*
Can disable filters by double-clicking or pressing Ctrl+T
*
When either the error display or the file display is full it is sent to the
log file
*
/z command line parameter to read parameters from a file
*
Grep-based regular expression matching
*
In-place editing of file specifications
*
Inclusion and exclusion lists for file processing
*
File groups allow better handling of complex file groupings
*
Subfolder recursion can now be specified on a per-file basis. Improved
performance of subfolder recursion
*
Improved detection for binary files - sample size can be adjusted, and binary
characteristics are better identified
*
Case-sensitive replace - if found text is uppercase, replacement can be
forced to all uppercase etc
*
New Export Filter to clipboard feature. Useful for documenting filters in
Word
*
Enhanced detail when printing filters
*
File associations for .dCv test files
*
Status dialog now shows separate error tab, and shows files generated by each
filter, with the old and new size of each with a comparison display for quick
checks
*
Install/uninstall feature. Creates Start Menu items, desktop icon, Send To
menu and file associations for .fll and .map files and for text files
*
Scratch pad area for test text
*
Trial run area for iterative testing
*
Improved error log for command line initiation
*
Pasted text is scanned for binary characters - and they are converted to
readable text. Pasted text is now checked to see if it has already been
converted
*
Heaps of useful sample filters
*
Window sizes now saved and restored. Minimum window sizes enforced
*
File list now supports including and excluding files, file groups, subfolder
inclusion/exclusion on a per-line basis
*
Scratch Pad area for saving chunks of text
*
New Edit Menu with Cut, Copy, Paste and Disable/Enable Filter
*
Settings now saved on a per-user basis
*
Improved descriptions in filter window
*
Improved performance. Better use of memory (now uses far less)
*
Scrolling on drag of filters in filter list
*
More filters and options available from command line
*
Now handles filters from future versions better
*
Printer setup menu item now doesn't get lost :-)
*
Fixed bug with "Base map on filter list" function

What's New in TextPipe v4.3
===========================
*
Filter to insert a column of text
*
Filter to convert to ranDOm CasE
*
Filter to extract URLs. Very useful for retrieving mailto, http, https, ftp
and gopher addresses
*
Task bar menu. Allows filters to be loaded and run without leaving external
programs such as Word, enabling conversion to occur seamlessly without
leaving the currently running program
*
All status windows from completed tasks are closed when starting a new task
*
Input filter new options:
  Input from file or clipboard
*
Output filter new options:
  Output to file or clipboard (independent of Input filter)
  change file extension
  change base directory on output
*
Filter to only show duplicate lines
*
Filter to reverse each line
*
Filter to centre justify lines
*
Filter to right justify lines
*
Filter to remove lines contained in an external file
*
Filter to only show lines contained in an external file
*
Filter to run external program. Very useful for user-written filters or for
features not yet incorporated into DropConvert
*
Filter to add comments to the filter list
*
Character maths filter now shows hex and binary equivalents for operand
*
File history (up to 5 filters) shown in file menu
*
Tool bar buttons for commonly-used filters
*
Floating hints now used in preference to status bar
*
Explorer menus for .fll file type
*
Removed evaluation edition restriction of 5 filters
*
Now uses InstallShield install/uninstall

See http://www.datamystic.com for more details on TextPipe

What's New in TextPipe – pending
===========================================

*
64 bit Windows Explorer shell extension! – for quick launching of filters.
*
Simultaneous search now far faster and supports 100,000+ search phrases
(exact match only).


Pro vs Standard version

 Pro vs Standard version
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Please see the latest version of this chart online at

http://www.datamystic.com/textpipe/pro_compare.html


Installation

 Installation
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe is simple to install. Simply double-click on the .exe file that you downloaded (usually textpipepro.exe, textpipestandard.exe, textpipeweb.exe etc).

The installation program creates uninstall entries under Control Panel:Add/Remove Programs. These allow TextPipe to be cleanly removed from your system.

Uninstall

Uninstallation is completely automatic.

To uninstall TextPipe, click the Start Menu, Settings, Control Panel, Add/Remove Programs. Select TextPipe from the list presented, and click Add/Remove. TextPipe will then be deleted.

The uninstall does not remove files modified or added by you, so any changes you have made to filter files (.fll) or maps (.map) will not be lost.


Unattended install

 Unattended install
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Unattended Installation

The Setup program accepts optional command line parameters. These can be useful to system administrators, and to other programs calling the Setup program.

/SP- Disables the This will install... Do you wish to continue? prompt at the beginning of Setup. Of course, this will have no effect if the DisableStartupPrompt [Setup] section directive was set to yes.

/SILENT, /VERYSILENT Instructs Setup to be silent or very silent. When Setup is silent the wizard and the background window are not displayed but the installation progress window is. When a setup is very silent this installation progress window is not displayed. Everything else is normal so for example error messages during installation are displayed and the startup prompt is (if you haven't disabled it with DisableStartupPrompt or the '/SP-' command line option explained above)

If a restart is necessary and the '/NORESTART' command isn't used (see below) and Setup is silent, it will display a Reboot now? messagebox. If it's very silent it will reboot without asking.

/NORESTART Instructs Setup not to reboot even if it's necessary.

/LOADINF="filename" Instructs Setup to load the settings from the specified file after having checked the command line. This file can be prepared using the '/SAVEINF=' command as explained below.

Don't forget to use quotes if the filename contains spaces.

/SAVEINF="filename" Instructs Setup to save installation settings to the specified file. Don't forget to use quotes if the filename contains spaces.

/DIR="x:\dirname" Overrides the default directory name displayed on the Select Destination Directory wizard page. A fully qualified pathname must be specified. If the [Setup] section directive DisableDirPage was set to yes, this command line parameter is ignored.

/GROUP="folder name" Overrides the default folder name displayed on the Select Start Menu Folder wizard page. If the [Setup] section directive DisableProgramGroupPage was set to yes, this command line parameter is ignored.

/NOICONS Instructs Setup to initially disable the Don't create any icons check box on the Select Start Menu Folder wizard page.

/COMPONENTS="comma separated list of component names" Overrides the default components settings. Using this command line parameter causes Setup to automatically select a custom type.


Online Web Purchase

 Online Web Purchase
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The commercial version of TextPipe can be purchased and received immediately on the Internet, eliminating shipping and handling costs. The latest price list can be downloaded from http://www.datamystic.com/order.txt

Direct Purchase URL: http://www.datamystic.com/buy.html

This purchase URL may be used to purchase all the following TextPipe licenses:

· TextPipe Standard
· TextPipe Lite
· TextPipe Pro
· TextPipe 5 User Pack
· TextPipe 10 User Pack
· TextPipe Educational Site Edition
· TextPipe Site Edition
· TextPipe Server Edition
· TextPipe Company Edition

See also

Purchase Information

Order Form

License Information


Order Form

 Order Form
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

DATAMYSTIC ORDER FORM

=====================

Prices current as at Monday, 9 January 2006

If you are using NotePad, please maximize the window before

printing.

All our products come with (unless stated below)

* One Year of FREE upgrades

* 30 days FREE Phone support

* Bonus additional software

* Unlimited Access to Technical Support web site and product FAQs

* Unlimited Access to Discussion Forums.

I N T E R N E T O R D E R S

-----------------------------

Please go to

http://www.datamystic.com/buy

to pay by credit card using the Internet.

U P G R A D E S

---------------

One year of FREE upgrades is included with each product. The upgrade price varies for

each product - please see our website for upgrade price details:

http://www.datamystic.com/buy

P H O N E O R D E R S

-----------------------

You can order by credit card over the phone if you do not have

Internet access.

USA Office: +1 724-850-8186

USA and Canadian customers: call 1-800-903-4152

German Office: +49-221-31088-20

Please have the Product Id (see the Product list below) and

your credit card ready.

F A X / M A I L O R D E R S

-----------------------------

You can order by fax or postal mail if you do not have Internet access. Please use

the order form below for:

* Purchase orders (commercial, government, educational etc)

* Fax orders

* Postal mail orders

* Cash payments

* Check payments

* All other non-internet orders

1. Please download the latest version (*) of this form from

http://www.datamystic.com/order.txt

(*) Prices may not be honored if you do not use the

latest version

2. Print out and complete this form

3. Fax or mail it to:

USA Office German Office

ShareIt! Inc. ShareIt!

PO Box 844 element 5 AG

Greensburg Vogelsanger Str. 78

PA 15601-0844 50823 Köln/Cologne

U.S.A. Germany

Tel. +1-724-850-8186 Tel: +49-221-31088-20

Fax. +1-724-850-8187 Fax: +49-221-31088-29

C O N T A C T D E T A I L S

-----------------------------

First Name: _________________________________________________

Last Name: _________________________________________________

E-mail: _________________________________________________

** Please do not forget to include your e-mail address **

*** We will use e-mail to communicate with you. ***

Company: _________________________________________________

Address: _________________________________________________

_________________________________________________

_________________________________________________

City: ___________________________________________

State: __________________________________________

Postal code: ____________________________________

Country: ________________________________________

Phone: _________________________________________________

Fax: _________________________________________________

How would you like to receive the registration key/full version?

(Please tick one)

[ ] E-mail (please ensure you have filled it in above)

[ ] Fax (please ensure you have filled it in above)

[ ] Postal mail (additional charge may be levied)

P A Y M E N T

-------------

How would you like to pay the registration fee? (Please tick

one)

[ ] Credit card (please fill out the details below)

[ ] Wire transfer

[ ] EuroCheque

[ ] Cash

[ ] US Check drawn on a US bank

Credit card information (if applicable)

[ ] Visa

[ ] Eurocard/Mastercard

[ ] American Express

[ ] Diners Club

[ ] Other: _________________________

Card Holder: ______________________________________________

Card Number: ____________________________ Expiry Date (mm/yyyy): ___/_____

Date / Signature: _________________________________________

P R O D U C T S

---------------

Please note: All prices are in USD. If you wish to order in Euros or another

currency, please indicate this when you order and will perform the appropriate

conversion.

** Please contact us for other licensing needs **

ID

DESCRIPTION AND PRICE

TextPipe Pro (fix text fast in web sites, database extracts and e-

text of unlimited size!)

Note: More features than Standard version. Please ensure

you are evaluating the Pro version before ordering

131686

131686

131686

1-2 Single User Licenses @ $299: $______

3-9 Single User Licenses @ $230: $______

10+ Single User Licenses @ $220: $______

__ x Server License @ EMAIL: $______

__ x Site License @ EMAIL: $______

__ x Enterprise License @ EMAIL: $______

TextPipe Standard

Note: Less features than Pro version. Please ensure you

are evaluating the Standard version before ordering

137374

__ x Single User License @ $169: $______

TextPipe Lite

Note: Less features than Standard and Pro versions.

Please ensure you are evaluating the Web version before

ordering

157097

__ x Single User License @ $40: $______

TextPipe Pro Run Time

Note: (the run time, read-only version of TextPipe Pro -

filters CANNOT be modified by clients)

147437

147438

__ x Single User License @ $149: $______

__ x Server License @ $249: $______

ResumePipe Subscription (convert emails and attachments to text

files) - includes TextPipe Pro filtering

Subscription based on number of resumes processed

200766

Resumes C/Res Total

1,000 $0.25 $ 250

2,000 $0.22 $ 440

5,000 $0.19 $ 950

10,000 $0.16 $ 1600

20,000 $0.13 $ 2600

50,000 $0.10 $ 5000

100,000 $0.07 $ 7000

200,000 $0.05 $10000

500,000 $0.04 $20000

$______

ResumePipe Unlimited Resume Model

137750

__ x Single User License @ $499: $______

DataPipe (interactive search and replace in databases)

138135

__ x Single User License @ $590: $______

__ x Site License @ EMAIL: $______

WordPipe Subscription Model (search and replace in multiple Word

documents)

Subscription based on number of documents MODIFIED - searching is

FREE

200779

Docs C/Doc Total

100 $0.50 $ 50

200 $0.45 $ 90

500 $0.40 $ 200

1000 $0.35 $ 350

2000 $0.30 $ 600

5000 $0.25 $1250

10000 $0.20 $2000

20000 $0.15 $3000

50000 $0.10 $5000

100000 $0.06 $6000

$______

WordPipe Unlimited Document Model

138134

__ x Single User License @ $499: $______

ExcelPipe Subscription Model (search and replace in multiple Excel

spreadsheets)

Subscription based on number of spreadsheets MODIFIED - searching

is FREE

203085

Docs C/Doc Total

100 $0.50 $ 50

200 $0.45 $ 90

500 $0.40 $ 200

1000 $0.35 $ 350

2000 $0.30 $ 600

5000 $0.25 $1250

10000 $0.20 $2000

20000 $0.15 $3000

50000 $0.10 $5000

100000 $0.06 $6000

$______

ExcelPipe Unlimited Spreadsheets Model

186919

__ x Single User Licenses @ $499: $______

PowerPointPipe Subscription Model (search and replace in multiple

PowerPoint presentations)

Subscription based on number of presentations MODIFIED - searching

is FREE

300010268

Docs C/Doc Total

100 $0.50 $ 50

200 $0.45 $ 90

500 $0.40 $ 200

1000 $0.35 $ 350

2000 $0.30 $ 600

5000 $0.25 $1250

10000 $0.20 $2000

20000 $0.15 $3000

50000 $0.10 $5000

100000 $0.06 $6000

$______

PowerPointPipe Unlimited Presentations Model

220500

__ x Single User Licenses @ $499: $______

TextPipe Engine Programmers DLL

132910

__ x per Application it is used in

Royalty Free @ $500: $______

AbusePipe spam complaint analyzer

188875

__ x Single User License @ $399: $______

EasyPattern Engine Programmers DLL

211244

__ x per Application it is used in

Royalty Free @ $199: $______

HookKeys Programmers DLL

TBA

__ x per Application it is used in

@ EMAIL: $______

DetachPipe (save, detach and delete email attachments)

146668

138333

166349

151810

1-2 Single User Licenses @ $50: $______

3-9 Single User Licenses @ $45: $______

10+ Single User Licenses @ $40: $______

__ x Home License @ $30: $______

__ x Server License @ EMAIL: $______

__ x Enterprise License @ EMAIL: $______

EmailPipe (converts emails to text files)

198245

__ x Single User License @ $499: $______

Time Zone Helper (show international working time overlap)

162196

__ x Single User License @ $20: $______

DirSize disk space monitor

132558

140086

140426

1-4 Single User Licenses @ $40: $______

5-9 Single User Licenses @ $36: $______

10+ Single User Licenses @ $34: $______

__ x Server License @ $349: $______

__ x Site License @ EMAIL: $______

DirDate

132564

1-4 Single User Licenses @ $20: $______

5-9 Single User Licenses @ $18: $______

10+ Single User Licenses @ $17: $______

Arc Menu (formerly Launch)

132561

157148

1-4 Single User Licenses @ $20: $______

5-9 Single User Licenses @ $18: $______

10+ Single User Licenses @ $17: $______

__ x Enterprise License @ EMAIL: $______

ZeroIn

100503

1-4 Single User Licenses @ $20: $______

5-9 Single User Licenses @ $18: $______

10+ Single User Licenses @ $17: $______

LFNit!

131928

1-4 Single User Licenses @ $40: $______

5-9 Single User Licenses @ $36: $______

10+ Single User Licenses @ $34: $______

BabyShield keyboard and mouse disabler

131903

__ x Single User License @ $9.99: $______

KeyboardLocker - let your child watch DVDs!

205713

__ x Single User License @ $19.99: $______

Clean 'n' Go mouse, keyboard and monitor cleaner

195484

__ x Single User License @ $9.99: $______

Coupon Code (if any): _____________

===================

Total: USD

===================


Purchase Information

 Purchase Information
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe is developed as a commercial product, and you may evaluate it to determine if it meets your needs. You must purchase TextPipe if you use it beyond the evaluation period or if you use any of the output for non-evaluation purposes. Use of TextPipe after the evaluation period is illegal and violates international copyright laws.

Benefits of Purchasing TextPipe

· 60 days FREE email support
· 60 days FREE updates
· Access to user support groups
· Notification of new releases, updates and upgrades
· Access to pre-release products. You will receive access to the newest tools created by DataMystic before they are released to the public.

Purchasing

There are two ways to purchase TextPipe:

· The easiest, quickest and most convenient way is to visit the TextPipe home page on the Internet at http://www.datamystic.com and use the online purchase facility. This facility is for credit card payment only.
· Mail or email the order form to DataMystic. The mailing address is contained in the form. The order form is contained in the help file and in ORDER.TXT
· You can also order over the phone using the details in the order form

See also

Online Web Purchase

Order Form

License Information


Other documentation

 Other documentation
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Contact Information

Other DataMystic Products

Shareware Vendor Information

Distribution Files

License Information

Corporate Modifications


Contact Information

 Contact Information
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Contact Information

For technical support, please see the support page.

For sales, please contact our sales department directly:

mailto sales@datamystic.com

For all other enquiries, please write to our general address:

mailto mail@datamystic.com

Phone/Fax

Phone - phonen1 +61-3 9913-0595 (9am-5pm Australian Eastern Standard Time, GMT+10 hours). Within Australia please call (03) 9913-0595

Fax - fax +61-3 8610-1234. Within Australia, please fax (03) 8610-1234.


Other DataMystic Products

 Other DataMystic Products
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Other DataMystic Products

DataMystic develops system utilities for power users, administrators and programmers. It has developed 14 software titles since 1990:

textpipe TextPipe | Automated search and replace and text processing tool
datapipe DataPipe | Uses TextPipe Pro to modify database records live in your databases
resumepipe ResumePipe | Extract emails and their attachments from Outlook, Lotus Notes, Applix or MAPI-compliant email systems to sequentially numbered files, and post-process using TextPipe Pro
detachpipe DetachPipe | Save, detach and delete email attachments from Outlook, Lotus Notes, Applix or MAPI-compliant email systems
wordpipe WordPipe | Perform multiple search/replaces in multiple MS-Word documents
excelpipe ExcelPipe | Perform multiple search/replaces in multiple MS-Excel spreadsheets
powerpointpipe PowerPointPipe | Perform multiple search/replaces in multiple MS-PowerPoint presentations
abusepipe AbusePipe | Detect IP address of Spam offenders from email complaints
emailpipe EmailPipe | Convert emails to numbered text files
timezone Time Zone Helper | Displays the working time overlap for people in different time zones
dirsize DirSize | Windows 95 command line utility to accurately show used and wasted disk space, sorted by total size, depth, wastage etc
zeroin ZeroIn | Fast CD replacement with spelling correction, history, aliases and long filename support
shuticon Arc Menu | Text-based, mouse-driven windowed menu system with many advanced capabilities
dirdate DirDate | Unique ability to change the date on folders
babyshield BabyShield | Temporarily disables the mouse AND keyboard while your pet or child taps on the keyboard
clipsize ClipSize | Displays memory occupied by the clipboard and allows it to be cleared easily
cleango Clean 'n' Go | Temporarily disables the mouse, keyboard or screen while they are cleaned.
lfnit LFNit! | Restores Windows 95 long filenames. Designed to allow continued use of old 16 bit applications (eg XTree, tape backup software, LapLink, PkZip etc) without losing long filenames

Contact Information


Shareware Vendor Information

 Shareware Vendor Information
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Vendor Information

If you are a

· Shareware distributor
· Disk vendor
· Web site
· CDROM publisher
· Magazine
· or any other organization or person

you may distribute the Evaluation Editions (*) of all DataMystic programs provided the following conditions are met:

1. You must inform the buyer that they are paying for your service, not for the programs, and that the programs must be purchased if they wish to continue using them after the evaluation period.
2. You must include all original files included with the program, and you must not modify these files in any way. If required, you may add new files of your own.
3. You must let us know by email if you include our software in your collection. We can then keep you informed of the latest version.

(*) All programs use the Help\About dialog or the -? Command line parameter to identify that they are an Evaluation Edition.

Contact Information


Distribution Files

 Distribution Files
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The TextPipe archive consists of the following files:

TEXTPIPE.EXE | The TextPipe program
*.FLL | Useful filters for common tasks
*.MAP | Useful maps for common tasks
DESC.SDI | Archive description file
FILE_ID.DIZ | Archive description file
ORDER.TXT | Order form
README.TXT | Readme file
TextPipe.HLP | Help file for 32-bit version
VENDOR.TXT | Vendor information
CORPORAT.TXT | Information about Corporate vesions
LICENSE.TXT | User license information
PRODUCTS.TXT | Information about other DataMystic products
WHATSNEW.TXT | What's new in this version of TextPipe

If any of these files are missing from your copy of TextPipe then you can obtain a complete version from the DataMystic Home Page on the Internet.


License information

 License information
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This program and the accompanying files and documentation are

Copyright (c) 1990- 2006 DataMystic

All rights reserved

This file contains our end user license. It also explains the terms of the Commercial License where they're

different from the Evaluation License.

If you have any questions, please contact us at

Address:

DataMystic

5 Bond St

Mt Waverley

Victoria 3149

AUSTRALIA

Phone:

(+61) 3 9913-0595 (GMT+10 hours)

Fax:

(+61) 3 8610-1234

Web:

http://www.datamystic.com

====================================================================

Plain English Evaluation License Agreement and Limited Warranty

1. SUMMARY. You may install this SOFTWARE on as many computers as your license allows, and no more:

You may access the software through a network, provided that you have obtained individual licenses

covering all computers that will run the software from the network. For instance, if six different

workstations will access the software on the network, you will need a six-user license, whether they use

the software at different times or concurrently.

* Home License/Personal Edition

May be used by one person in a home environment. CANNOT be used in government, business or any

organization.

* Single User License/Desktop Edition/Single User Pack

May either be used by a single person who uses the software himself on one or more non-server class

computers, or installed on a single non-server class computer and used by multiple people, but not both.

* 10 Days Only License (TextPipe Pro only)

May be used for 10 unique calendar days (consecutive or non-consecutive), as determined by our

license server.

* Server License

May be installed on ONE (1) server class computer ONLY

* Site License/Educational Site License

May be installed on UP TO 500 computers of server class or non-server class at one site ONLY

* Enterprise License/Company Edition/University Edition

May be installed on UP TO 2000 computers of server class or non-server class directly owned by a

Company or University ONLY (but not on client's computers)

This is a legal agreement between you and DATAMYSTIC. By using the SOFTWARE you are agreeing to be

bound by the terms of this Agreement. Do not use the SOFTWARE if you do not agree to the terms of this

Agreement.

2. DEFINITIONS. In this agreement:

*

"SOFTWARE" means the computer program you have purchased or obtained for evaluation,

including the program, associated documentation and other materials;

*

"Evaluation Copy" means a copy of the SOFTWARE distributed by us or by our authorised

agents for evaluation purposes and which is described as an "Evaluation Edition" or a

"Shareware Edition" by the SOFTWARE's "About" screen;

*

"Commercial Copy" means a copy of the SOFTWARE purchased from us or from a dealer,

and is NOT described as an Evaluation Copy by the SOFTWARE's "About" screen. The actual

description may include "Desktop Edition", "Single User Edition", "Personal Edition", "Server

Edition", "Site Edition" or "Company Edition" or other variations depending on the version

purchased;

*

"you" means the end user of the SOFTWARE. This can be either an individual or an entity;

*

"server class" means a computer designated as a file server, database server, application server or

other kind of server;

"desktop class" or "non-server class" means a computer where only one person uses it at a time;

*

"DATAMYSTIC" means DataMystic, an Australian corporation (address shown above).

3. GRANT OF LICENSE. DATAMYSTIC grants you the right to INSTALL this SOFTWARE on as many

computers as you have purchased a license for. You may NOT install it on computers that you do not have a

license for.

The SOFTWARE is "installed" on the computer when it is loaded into permanent memory (e.g. hard disk, CD-

ROM, or other storage device) of the computer.

4. COPYRIGHT. The SOFTWARE is owned by DATAMYSTIC and is protected by copyright laws and

international treaty provisions and all other applicable national laws. Therefore, you must treat the SOFTWARE

like any other copyrighted material (e.g. a book or musical recording) except that you may make one copy of the

SOFTWARE solely for backup or archival purposes. You may not copy the user documentation provided with

the SOFTWARE.

5. OTHER RESTRICTIONS. You may not rent, lease, sell, or attempt to rent, lease or sell the SOFTWARE.

You may transfer the SOFTWARE and accompanying written materials on a permanent basis provided you

retain no copies and the recipient agrees to the terms of this Agreement.

You must not copy, translate, adapt, modify, alter, decompile, disassemble, or otherwise reverse engineer or

create any derivative work of the SOFTWARE, merge the SOFTWARE or any part thereof with any other

software or change any SOFTWARE in whole or in part, nor allow any other party to do so. You must not alter

or remove any copyright or other intellectual property notifications applied to the SOFTWARE.

You acknowledge that the SOFTWARE and the copyright, and other intellectual property rights of whatever

nature in the SOFTWARE are not your property and contain valuable proprietary information and trade secrets

and that this license confers no title or ownership in the SOFTWARE and no rights in any associated source

code, and is not a sale of any rights in the SOFTWARE.

6. EVALUATION PERIOD. You may use an Evaluation Copy of the SOFTWARE for up to 30 days in order to

determine whether it meets your needs. While you are evaluating the SOFTWARE, you may use it on as many

computers as are required to perform your evaluation. Your evaluation period begins when you first install the

SOFTWARE on one or more computers for evaluation purposes. Once the evaluation period is over you MUST

PURCHASE THE SOFTWARE or REMOVE IT FROM ALL COMPUTERS. If you use the SOFTWARE for non-

evaluation purposes DURING your evaluation you MUST PURCHASE THE SOFTWARE.

7. MAKING AND DISTRIBUTING EVALUATION COPIES OF THE SOFTWARE. You may make copies of your

Evaluation Copy of the SOFTWARE to give to others, as long as you include all of the files that you originally

received with your Evaluation Copy. When you give an Evaluation Copy of the SOFTWARE to another person,

you agree to inform them that their copy is to be used for a time-limited evaluation period, and that they must

purchase a Commercial Copy if they continue to use the SOFTWARE once the evaluation period has ended.

You agree not to sell Evaluation Copies of the SOFTWARE or distribute them to others for any kind of

compensation or fee unless the fee solely covers your costs in providing it to the end user.

8. OPERATION OF THE SOFTWARE. The SOFTWARE offers powerful capabilities. If you do not follow our

instructions, or if you use the SOFTWARE improperly, you can destroy files or cause other damage to your

software and data. You assume full responsibility for the selection and use of the SOFTWARE to achieve your

intended results.

9. LIMITED WARRANTY. The following limited warranty applies to Commercial Copies of the SOFTWARE.

DATAMYSTIC warrants that for a period of ninety (90) days from the date of receipt of the SOFTWARE that (i)

the SOFTWARE will perform in substantial conformance with the related documentation supplied with the

SOFTWARE; and (ii) the media on which the SOFTWARE is distributed is free from defects in materials and

workmanship. If the SOFTWARE does not conform to these limited warranties DATAMYSTIC will, at its option,

either refund the amount you paid for the SOFTWARE or provide you with the corrected item at no charge

provided that the defective item is returned to DATAMYSTIC within ninety (90) days from the date of your

receipt of the SOFTWARE. Any replacement SOFTWARE will be warranted for the remainder of the original

ninety (90) day warranty period or thirty (30) days from the date you receive the replacement, whichever is

longer. Any misuse, accident or unauthorized modification of the SOFTWARE will void these limited warranties.

This warranty does NOT apply to Evaluation Copies of the SOFTWARE.

10. INDEMNITY. DATAMYSTIC shall defend, at its expense, and pay the cost and damages incurred or

awarded as a result of any action brought against you based on an allegation that the SOFTWARE infringes any

third party Intellectual Property Rights provided: you promptly notify DATAMYSTIC in writing of any such action

or allegation of infringement; DATAMYSTIC has complete control of the defense of any such action and all

negotiations for its settlement or compromise; and, you at no time admit liability or otherwise settle or

compromise or attempt to settle or compromise the claim or action except under the express instructions of

DATAMYSTIC.

In addition to DATAMYSTIC's obligations under the above paragraph, if, in DATAMYSTIC'S opinion, the

SOFTWARE is likely to become the subject of a claim of any infringement, DATAMYSTIC shall, at its sole

option and expense: (a) procure for you the right to continue using the SOFTWARE; (b) replace or modify the

SOFTWARE so that it becomes non-infringing; or, at DATAMYSTIC'S sole discretion, if (a) and (b) are not

feasible, DATAMYSTIC shall accept the return of such infringing SOFTWARE and reimburse you for the

purchase price of such infringing SOFTWARE.

11. DISCLAIMERS AND LIMITATIONS. And now the legal language we couldn't avoid:

DISCLAIMER FOR EVALUATION COPIES: YOU ARE NOT PAYING FOR THE TRIAL USE LICENSE FOR

YOUR EVALUATION COPY OF THE SOFTWARE. ACCORDINGLY THE EVALUATION COPY IS PROVIDED

"AS-IS" AND WITHOUT WARRANTY OF ANY KIND.

DISCLAIMER FOR COMMERCIAL COPIES: Except for the limited warranty stated above, we disclaim any and

all other warranties, express or implied, oral or written, including any implied warranties of merchantability or

fitness for a particular purpose. The limited warranty above gives you specific legal rights, but you may have

other rights, depending on where you purchased the SOFTWARE.

LIMITATION OF LIABILITY. IN NO EVENT SHALL EITHER PARTY BE LIABLE FOR ANY INCIDENTAL,

CONSEQUENTIAL, OR PUNITIVE DAMAGES WHATSOEVER ARISING OUT OF USE OF THE SOFTWARE

OR ONE PARTY'S RELATIONSHIP WITH THE OTHER PARTY. SOME COUNTRIES OR STATES DO NOT

ALLOW EXCLUSION OR LIMITATION OF LIABILITY FOR INCIDENTAL OR CONSEQUENTIAL DAMAGES,

THEREFORE THE ABOVE LIMITATION MAY NOT APPLY TO YOU.

12. This agreement is to be interpreted under the laws of the state of Victoria, Australia.

END OF LICENSE AGREEMENT


Corporate Modifications

 Corporate Modifications
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Corporate Modifications

Corporate and business users are welcome to approach DataMystic for modifications specific to their needs. We can easily incorporate any changes you desire. Please send us specifications and we will send you a quote. Please also include details such as a contact name, phone and/or address, plus any other details you think are relevant.

Contact Information


How To Use TextPipe

 How To Use TextPipe
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Using TextPipe is simple:

1. Choose the filters you wish to apply.

Filter can be chosen from the Convert, Add, Remove, Email, Special and Map menus. Most filters can be further customized in the right hand area of the screen - just click on the filter you wish to customize.

1. Choose the files you wish to process (or select the clipboard).

You can drag and drop files from Explorer onto TextPipe's window (anywhere) and they will be placed in the file list. Folders dragged will have "*.*" added to their name - meaning all files in the folder will be processed. You can process the clipboard if you select the Input filter (at the very top of the filter list) and then select Input from Clipboard.

1. Click the "Go" button to start conversion.

If you wish to override the Input/Output filter settings and instead, take input from the clipboard and then send output to it, click the "Clipboard" button.

While the conversion is in progress, you can click the "Stop" button to cancel it, or you can use the main window to build up another conversion job.

The Clipboard button is always enabled, even if the clipboard is empty. The Go button is only enabled if the file list contains at least one file, or if the input filter specifies that Input comes from the Clipboard.

See also

Drag and Drop

File Lists

Filter Lists


Filters to Apply Tab

 Filters to Apply Tab
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The Filters to Apply tab of the main TextPipe window controls the operations that are performed on each file.

All Available Filters is a categorized view of all TextPipe's filters. It also contains links to wizards, useful documentation and other options. To add one of these filters, either double click it, or drag and drop it to the right hand My Filters List.

My Filter List is a list of the filters that will be applied to your text; each is briefly described. For more detail, click the node, and the settings will be shown on the far right. The Num column shows the order that filters will be applied in. You can control the order by dragging and dropping nodes, or by using the Up/Down buttons at the base of the tree.

Double clicking on a filter toggles it between enabled and disabled state. A filter that has been disabled does not take part in filtering operations, and its icon is crossed out.

Adding filters

To add a new filter to the list:

1. Place the cursor ABOVE the position where you would like the new filter. The new filter will be added below the cursor
2. Double-click or drag and drop the filter from the All Available Filters view
3. The new filter will be added below the cursor
4. Edit the filter's properties on the right hand side of the window

Remove

To remove a filter, select it and press [Delete] or click [Remove].

Remove All

To remove all filters, click [Remove All], which will then ask for confirmation.

Drag and Drop

Filters can be dragged and dropped within the filter list to change their position. Filters cannot be dragged to a position prior to the Input Filter or after the Output Filter. During drag and drop, moving the mouse above or below the list causes the list to scroll. The further the mouse is away from the list, the faster it will scroll.

To create a sub filter, drag a filter onto the text of another filter node (the mouse cursor will display a barred circle if the filter does not accept sub filters). An alternative to positioning the mouse in the right hand side of the tree view is to hold down the [Shift] key while you drag. You can also hold down the [Alt] key when you add a new filter and it will be added as a sub filter of the current filter (if allowed).

Watch the status bar for information of whether TextPipe is going to simply swap filter positions or if it will create a sub filter.

Right-click menu

The tree view has a context menu which can be used to Enable or Disable a filter, delete it, Cut, Copy or Paste a filter or group of filters, and perform partial trial runs.

Partial Trial Runs

You can perform a Trial Run using the text in the Trial Run area. This is initiated using the [Trial Run] button. You can also perform a partial trial run by right clicking on a filter, and choosing a trial run of the selected filter or all filters up to that point. This is a very handy way of debugging filters.

Keyboard commands

Alt+Up | Moves the selected filter up one position
Alt+Down | Moves the selected filter down one position
Alt+(add a new filter) | If allowed, TextPipe adds the new filter as a sub filter of the currently selected filter.
Shift+(add a new filter) | TextPipe adds the new filter at the parent level of the current filter.
Shift+(drag a filter) | Tries to add the dragged filter as a sub filter regardless of whether the mouse is in the right hand half of the tree view or not.

Command Line

A filter list file can be opened on the command line by using the /F=<filename> option.

See also

Drag and Drop

File Lists

Sub filters


Sub filters

 Sub filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Sub filters allow a restricted part of the entire text to be operated on by a group of filters without effecting the entire text. For example, a Restrict to delimited fields (CSV, Tab, Pipe etc) filter can pick out a range of CSV fields, and then a search/replace filter can operate JUST on the text restricted.

Creating a sub filter

To create a sub filter, simply drag a filter onto the text of another. If the filter supports sub filters (not all do), when you drop the filter it will be dropped as a sub filter. If you drag the filter to an area outside of the destination filter's text, you can control whether to add it above or below the destination filter. An alternative to positioning the mouse in the right hand side of the tree view is to hold down the [Shift] key while you drag. You can also hold down the [Alt] key when you add a new filter and it will be added as a sub filter of the current filter (if allowed).

Watch the status bar for information of whether TextPipe is going to simply swap filter positions or if it will create a sub filter.

Keyboard commands

Alt+Up | Moves the selected filter up one position
Alt+Down | Moves the selected filter down one position
Alt+(add a new filter) | If allowed, TextPipe adds the new filter as a sub filter of the currently selected filter.
Shift+(add a new filter) | TextPipe adds the new filter at the parent level of the current filter.
Shift+(drag a filter) | Tries to add the dragged filter as a sub filter regardless of whether the mouse is in the right hand half of the tree view or not.

Other examples

Delete 11 lines starting at line 3

Hex encode tab fields 3 to 7

Many different filters support sub filters.

Note - sub filters treat each section of restricted text as a complete file.

See also

Restrict to delimited fields (CSV, Tab, Pipe etc)

Restrict line range

Restrict column range

More restriction filters

Search/replace

Add left margin

Add right margin

Add header

Add footer

Insert column

Comments - not really a sub filter, but can be used to group filters together


Files to Process Tab

 Files to Process Tab
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Files to Process Tab

The Files to Process tab of the main TextPipe window allows you to configure the list of files to be processed. This tab is only used if the Input Filter is set to process files.

Specify files using a List File

You can load a list of files by specifying a list filename. This should be a text file with one file or one file spec per line.

Specify files with the Files Grid

The columns widths can be resized by dragging the column separator in the header.

Filename or Folder Name Column

The Filename column lists both individual filenames and wildcard specifications for files to be matched. e.g. c:\windows\*.ini will match all files with an extension of .ini in the c:\windows folder. If the filename doesn't contain wildcards (* or ?), then double clicking it or clicking the [View] button when the row is selected will open the file in its associated application. Single clicking the filename after it has been selected allows the row to be edited, and also shows an ellipsis button, which when clicked, allows you to browse for files.

Multiple file specifications can be separated with a semi-colon e.g.

c:\INetPub\*.htm;*.txt;*.asp

You can specify an internet link to download and then process by specifying the link e.g. http://www.hotmail.com. If you do not specify an output folder or merged output filename (or the clipboard) then each downloaded url is saved in the TEMP folder specified in the environment. The leading http:// part of the filename is stripped in this process.

Clicking on the Filename column heading sorts the file list. Double clicking on a row in the file list opens the file or folder.

Note - TextPipe allows files to be specified more than once. This can be useful when a file needs to be appended multiple times.
Note 2 - Because of its use in separating filespecs, filenames in the grid cannot contain a semicolon (;). You can workaround this by using a filespec that replaces the ; with a ? character.
Note 3 - both the * (match a group of characters) and ? (match a single character) wildcards are supported.

Subfolders Column

The setting in the Subfolders column determines whether TextPipe scans subfolders below the given location. For example, if the filename column contained c:\Program Files\*.txt, then setting the Subfolders column to None will only process *.txt files in this folder alone. Setting the Subfolders column to All will process all *.txt files in every subfolder below c:\Program Files, such as c:\Program Files\Microsoft Office, c:\Program Files\Microsoft Office\Templates etc. Single clicking the cell allows it to be edited, where it provides a drop down list with values All, None, 1 level, 2 levels etc.

Action Column

The setting in the Action column determines whether TextPipe adds this file specification to the list, Excludes matching files already on the list, or simply Ignores the row completely (useful for testing). For example (as above), if the file list contained an entry to include *.txt in all subfolders, and below it an entry to Exclude all entries matching 'Readme.txt' from all subfolders, then TextPipe will look at the file list it has gathered so far, and remove all matching entries eg.

Sample File List After 'Include *.txt in all subfolders'

d:\test.txt

d:\Readme.txt

d:\winnt\readme.txt

d:\winnt\modemDet.txt

d:\winnt\setuplog.txt

Sample File List After 'Exclude Readme.txt in all subfolders'

d:\test.txt

d:\winnt\modemDet.txt

d:\winnt\setuplog.txt

Exclude entries are shown in Red.

So:

Include = Add

Exclude = Subtract

Disable = Don't process this line at all

TextPipe also allows files to be removed from the list based on a Perl pattern or EasyPattern. This is very handy for removing files such as _vti_* files from Microsoft Frontpage web sites - these file should not be processed.

A perl pattern of

_vti_

is all that is required to remove all files that contain '_vti_' anywhere in the title.

Right click - Copy to Trial Input

Right clicking on a file shows a menu - allowing you to copy the selected file into the Trial Input area. If the file you click on has wildcards then TextPipe will pick the first file that matches the wildcards. Note that TextPipe ignores the settings of the Subfolders and Action columns.

Analyse

Attempts to detect the file encoding. Encodings detected include:

Unicode encoding UTF-32BE
Unicode encoding UTF-32LE
Unicode encoding UTF-16BE
Unicode encoding UTF-16LE
Unicode encoding UTF-16BE XML
Unicode encoding UTF-16LE XML
Unicode encoding UTF-8 XML
Unicode encoding UTF-8
EBCDIC XML
Unicode UTF-32BE
Unicode UTF-32LE
Unicode UTF-16BE
Unicode UTF-16LE
Unknown - ASCII or ANSI

Hex Dump

Displays a hex dump of the first 200 characters of the file. This is very handy for determining the file format.

Add Folder

Clicking the [Add folder] button displays a browse folder dialog, which lets you select a folder to be processed. When the folder is added, by default, '*.*' is appended to the folder name to indicate that all files in the folder will be processed. The new row containing the added folder defaults its Subfolder and Action columns from the row above.

Insert Row

Inserts a blank new row below the current row.

Copy Row

Inserts a new row below the current row that is an exact copy of the current row.

View

If the selected row's filename doesn't contain wildcards (* or ?), then double clicking it or clicking the [View] button when the row is selected will open the file in its associated application.

New Group

Adds a new file group below the current row. A file group limits the entries affected by Exclude entries. Exclude entries only act on files previously added within the current group, rather than affecting all files added so far.

Groups must have a colon ':' as the first character of their name, but the name is otherwise unrestricted.

Open File List

TextPipe can read the file list from a file. This means that you can easily create a list of files to process for various tasks, and open it when required rather than go through the laborious task of adding the files to the list each time.

The file consists of a single file specification per line eg.

c:\my documents\*.txt

c:\bp\*.pas

d:\job_ads\*.*

To open a file list, click on the open icon on the files tab. An open dialog box appears. Select one or more files to open and click on the Ok button.

Save File List

The currently displayed file list can be saved to a file by clicking on the Save icon on the Files tab. A save dialog appears and an existing filename can be selected or a new one specified.

Paste File List

Click the paste icon to paste a list of filenames from the clipboard.

Filter by Date

You can exclude files that are older or newer than a given date, entered either via a calendar for absolute dates, or by specifying the number of days.

Filter by Size

You can exclude files that are bigger or smaller than a given size, entered in bytes.

Remove

To delete one or more entries in the list, select them and and press [Delete] or click [Remove].

Remove All

To remove all entries, click [Remove All]. You must confirm this action.

Drag and Drop

Drag and drop files from Windows Explorer either onto TextPipe's main window or onto its icon (the Files tab does not have to be visible).

Command Line

Provide the names of the files you wish to process on the command line, either by name (using wildcards) or in a list file. A file list can be opened on the command line by using the /F=<filename> option.

How filenames are processed

TextPipe processes the Files to Process list row in order, so if you had listed

c.txt

b.txt

a.txt

then the files would be processed in the displayed order - c,b,a.

TextPipe does not sort the list of files to process (but you can sort the list by clicking the Filename column heading). In addition, if you use a wildcard, TextPipe retrieves files based on the Windows directory record order, which is generally unsorted.

The status tab always reflects the actual order that files are processed.

If a file matches a wildcard twice, it will be processed twice.


Comments Tab

 Comments Tab
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The Comments Tab is no longer used. Instead, comments may be placed freely throughout the filter list. Any old comments found when a filter is loaded are converted to a comment filter at the top of the filter list.

See also

Drag and Drop

Filter Lists


Scratch Pad Tab

 Scratch Pad Tab
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The Scratch Pad Tab is only available when the Edit\Preferences\Show Scratch Pad option is enabled. Also, if you load a filter that had Scratch Pad text, the tab is displayed as before.

See also

Drag and Drop

Trial Run Tab

Filter Lists


Trial Run Tab

 Trial Run Tab
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The Trial Run tab of the main TextPipe window gives you a test bed for trying out filters. The Trial Run area is used to test a filter repeatedly on small sets of text. The Trial Run input text is saved with each filter (the Trial Output is not).

Clicking the Trial Run button performs a Trial Run.

Important Notes

1. The Trial Input and Output areas do NOT understand encodings other than ANSI and Unicode 16LE. You CANNOT use a Trial Run to test conversions between different encodings - you must use input to and from a file, and view the resulting file (e.g. in Notepad)
2. The Trial Input and Output do NOT interpret any special character codes or macros or global variables
3. End-of-line characters - the Trial Run function can give slightly different results to those encountered when actually processing a file. The difference is because the Trial Input field automatically converts line feed characters into DOS line feed characters (\r\n). This means that while text in the Trial Input field may process correctly, the results from processing a file may be different if you are processing files with Unix or Macintosh line feed characters. The best way to avoid these problems is to include a convert EOL characters filter at the very start of your filter list, converting from Auto to DOS.

Extra Options

Both the Trial Input and Output field support right-click context menus for saving, loading etc.

Treat Trial Input as Unicode (UTF-16LE) instead of ANSI

When checked, the text in the Trial Input area is filtered as Unicode (i.e. double-byte UTF-16LE). When this flag is unchecked, Unicode text is first converted to ANSI text before it is processed. Normally when this setting is checked, Treat Trial Output as Unicode is also checked.

Treat Trial Output as Unicode (UTF-16LE) instead of ANSI

When checked, the text sent to the Trial Output area is assumed to be Unicode (i.e. double-byte UTF-16LE). Only use this setting if

1. The Trial Input is Unicode (see the setting above)

OR

1. You are using filters to convert text to Unicode form (UTF-16LE).

Synchronize Scrolling

When checked, moving the scroll bar of the Trial Input area moves the Trial Output scroll bar as well. This is handy for viewing filter results.

Move Trial Output to Trial Input

Identical to an item in the Tools Menu, this moves the text in the Trial Output area to the Trial Input area.

Open Diff Tool

Opens the Diff Tool configured in Edit\Preferences.

See also

Drag and Drop

Filter Lists


Format Specifiers

 Format Specifiers
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Format strings passed to the string formatting routines contain two types of objects--plain characters and format specifiers. Plain characters are copied verbatim to the resulting string. Format specifiers fetch arguments from the argument list and apply formatting to them. In the case of the Add Line Numbers filter, the only argument is the current line number.

Format specifiers have the following form:

"%" [index ":"] ["-"] [width] ["." prec] type

A format specifier begins with a % character. After the % come the following, in this order:

· An optional argument index specifier, [index ":"]
· An optional left justification indicator, ["-"]
· An optional width specifier, [width]
· An optional precision specifier, ["." prec]
· The conversion type character, type

The following table summarizes the possible values for type:

d Decimal. The argument must be an integer value. The value is converted to a string of decimal digits. If the format string contains a precision specifier, it indicates that the resulting string must contain at least the specified number of digits; if the value has less digits, the resulting string is left-padded with zeros.

e Scientific. The argument must be a floating-point value. The value is converted to a string of the form "-d.ddd...E+ddd". The resulting string starts with a minus sign if the number is negative. One digit always precedes the decimal point.

Conversion characters may be specified in uppercase as well as in lowercase--both produce the same results.

For all floating-point formats, the actual characters used as decimal and thousand separators are obtained from the DecimalSeparator and ThousandSeparator global variables.

A width specifier sets the minimum field width for a conversion. If the resulting string is shorter than the minimum field width, it is padded with blanks to increase the field width. The default is to right-justify the result by adding blanks in front of the value, but if the format specifier contains a left-justification indicator (a "-" character preceding the width specifier), the result is left-justified by adding blanks after the value.

An index specifier sets the current argument list index to the specified value. The index of the first argument in the argument list is 0. Using index specifiers, it is possible to format the same argument multiple times. For example "Format('%d %d %0:d %d', [10, 20])" produces the string '10 20 10 20'.


Macros and Global Variables

 Macros and Global Variables
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Macros allow you to insert information that varies while the filter runs - e.g. the name of the current file being processed. Macro names are not case sensitive.

Macro Name | Description
@inputFilename | Replaced by the name of the current file being processed. The drive and path section of the filename is removed. Usually used with the Add text at start of file filter to place the name of the file being processed at the beginning, especially if output is being merged to one file or to the clipboard e.g. \013\010-- This is file @inputFilename --\013\010
@fullInputFilename | Same as @inputFilename, except it includes the complete path
@inputPath | The path to the input file, with a trailing slash e.g. 'c:\my files\'
@inputExtension | The extension of the input file e.g. '.txt'
@fullOutputFilename | The complete path of the current output file. Note that this macro may be blank if you are processing the Trial Run area or Clipboard, and the result returned may be incorrect if your filters split or merge files.
@clipboard | Inserts the contents of the Windows clipboard
@filterFilename | Inserts the name of the current filter (if it has been saved)
@filterPath | Inserts the path of the current filter (if it has been saved)
@time | Inserts the time in the format HHMMSS
@date | Inserts the date in the format YYYYMMDD
@datetime | Inserts the date and time in the format YYYYMMDDHHMMSS
@randomdigit | Inserts a random digit from 0-9. For two digits, use @randomdigit@@randomdigit
@randomletter | Inserts a randomletter from A-Z. For two letters, use @randomletter@@randomletter
@ENV_varname | Where varname is the name of an environment variable, this inserts the value of a environment variable e.g. @ENV_OS inserts the text 'Windows_NT' on an NT system. See below for 'what are environment variables'
@global_variable_name | Where global_variable_name is the name of your global variable, set using the Capture Text filter, the Scripting filter, or by using the Named Variable capture of a perl or EasyPattern search/replace filter. Global variables are available to every filter in the filter list, so if there is any chance of a conflict you need to name them differently. This is particularly so if you link filters that contains Scripts that use global variables.

All fields that support macros have a "context" or right-click menu that allows them to be entered without remembering the exact code.

Special requirements | Replace With text: | With Macro Values | Results in:
If you need to put an @ in the text, simply double it i.e. @@. | hello@@ | N/A | hello@
If you need to include the literal name of a macro, just double the @ | @@randomdigit | @randomdigit=7 | @randomdigit
If you need to place letters or digits immediately after a macro name, end the macro name with @. | @randomdigit@@randomdigit | @randomdigit=7, @randomdigit=2 | 72
 | @randomdigit@@@ | @randomdigit=7 | 7@

What are Environment Variables?

Environment variables are part of Windows (DOS), to see their values go to a cmd.exe prompt and type

set

More info - please see http://en.wikipedia.org/wiki/Environment_variables

The following filters allow the use of macros to insert predefined text.

All Search/Replace filters

Add text at start of file

Add text at start of lines

Add text at end of lines

Add text to end of file

Add Column of Text


Drag and Drop

 Drag and Drop
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Files or folders dropped from Windows Explorer onto TextPipe's window are automatically placed in the files list. Folders have "*.*" automatically appended, so all files in that folder will be processed.

If you drag and drop a TextPipe .fll filter file it will automatically be opened.

See also

File Lists

Filter Lists


Popup menus

 Popup menus
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Popup menus (context menus)

All memo fields (multi-line text fields) and most edit fields support popup menus. Right click in the field to active the menu. TextPipe lists the applicable commands, and dims any commands that are not possible (e.g. if the clipboard is empty, is makes no sense to allow Paste).

Insert special character

TextPipe uses special notation to define characters that cannot be entered from the keyboard, such as a form feed, represented by '\f'. This menu gives you a list of options to insert a variety of special characters in the field. Depending on the type of field (e.g. Exact match, Pattern match), different characters will have a special meaning, and so the menu that is shown will change.

Insert control character

These menu items are used to insert control characters directly, without using special codes to represent them e.g. when a tab is inserted, an ASCII code 9 is inserted, not \t.

Insert pattern

Fields that support pattern matching will also display a submenu to help you quickly enter patterns for common situations.

Insert custom pattern

Fields that support pattern matching will also display a submenu allowing user-defined patterns to be entered. For more details on setting this up, please see defining custom patterns.

Insert macro

For edit fields that support macros (@inputFilename etc), this menu will automatically add one of the supported macro names.

Escape special characters

As mentioned above, TextPipe uses special notation to define characters that cannot be entered from the keyboard. Use this command to ensure that TextPipe cannot misinterpret any control characters that you may inadvertently have in the edit field e.g. backslash characters are doubled \ -> \\

Allow extra spaces

This command converts the selected text of a search pattern (or all text if there is no selection) into one capable of absorbing extra spaces or tabs found in the search string. This can be very handy for matching in the HTML documents of a website where the HTML editor may have inserted extra spaces or tabs.

This command is only available when the selected field is the Search field of a search pattern.

Allow extra line feeds and spaces

This command converts the selected text of a search pattern (or all text if there is no selection) into one capable of absorbing extra spaces, tabs or line feeds found in the search string. This can be very handy for matching in the HTML documents of a website where the HTML editor may have inserted extra spaces, tabs or line feeds.

This command is only available when the selected field is the Search field of a search pattern.

Allow RTF tags

This command converts the selected text (or all text if there is no selection) into one capable of absorbing RTF tags. RTF tags are present in a .RTF (Rich Text Format) files. This facility is almost essential for searching and replacing inside RTF files.

The Allow RTF tags feature is only designed for use with text that consists of normal words - not with text that already contains a pattern.

This command is only available when the selected field is the Search field of a search pattern.

Find

The Find command searches for text in the field.

Find next

The Find Next command searches for the next occurrence of the previous text searched for. TextPipe will beep if there is no next occurrence, and if Find Next is activated again it will recommence the search at the beginning of the field.

Word wrap

Toggles word wrapping for the memo control. This property is not saved - it just makes it easier to view text.

Show hex dump

Shows a hex dump of the selected text or the entire text if there is no selection. This is handy to find out exactly what control characters are present in a control.


Pattern matching

 Pattern matching
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict
· Try EasyPatterns for an easier pattern matching experience
· For more practical examples, take a look at the section, Examples Using Regular Expressions.
· Perl/PCRE pattern matching reference
· For further reference, we recommend Mastering Regular Expressions: Powerful Techniques for Perl and Other Tools (O'Reilly Nutshell) -- Jeffrey E. Friedl (Editor), Andy Oram (Editor); Paperback

Why do we need patterns?

We need patterns to describe text that is inexact, such as 'all words starting with s', 'all words starting with t and ending with m', or 'all 9 digit numbers'. We might be able to list all the possible variations, but it would be impractical and annoying.

We use a pattern matching language to describe a pattern. A pattern matching language uses symbols to describe both normal characters (that are matched exactly) and meta characters (that describe special operations, such as alternatives and repeating sections). Patterns are almost always described using what is called a regular expression, so called because the pattern matching language fulfills some mathematical properties that we don't need to worry about. Patterns can become very complicated very quickly, so let's start with some simple examples.

The text below refers to Perl-style patterns. For an easier-to-use pattern matching experience, try EasyPatterns.

We've all done a search and replace where we don't care about the case of the search string (Case Sensitive is Off). For example, if I search for 'Car', I also want to search for 'car, 'CAR' and any other case variations. We are implicitly searching for an upper or lower case 'C', followed by an upper or lower case 'A' followed by an upper or lower case 'R'. This is an example of a simple pattern.

In a pattern matching language, we could express this as follows:

[Cc][Aa][Rr]

Each set of square brackets is a character class, meaning 'find one of any of these characters' - a 'C' or 'c', then a 'A' or 'a', then a 'R' or 'r'. If we changed this pattern to

[BbCc][Aa][Rr]

then we also could find 'Bar', 'bar' and 'BAR' (and others). Of course, if we just toggled the Case Sensitivity flag then it wouldn't matter if we used

[bc][a][r] or [BC]AR or [bC]Ar or even [bc]ar

Note that the letters outside of the square brackets simply represent themselves - they are not special.

Here are some more character classes:

[aeiou] | matches any lower case vowel
[^aeiou] | matches any character that is NOT a lower case vowel
[0123456789] | matches any digit
[^0123456789] | matches any character that is NOT a digit
] | NOT special, NOT the end of a character class when on its own
[]abc] [abc\]] [^]abc] | matches a, b, c and ] matches a, b, c and ] matches everything except a, b, c and ] If a closing square bracket is required as a member of the class, it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash. It is not possible to have the literal character "]" as the end character of a range.
[0123456789^] [\^0123456789] | Matches 0 to 9 and ^ Matches 0 to 9 and ^ If a circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash.
[0-9] [d-m] [0-9a-zA-Z] [-0-9] [0-9-] [\000-\037] | Matches 0 to 9 Matches d to m Matches 0 to 9, a to z, A to Z Matches 0 to 9 and - Matches 0 to 9 and - Matches ASCII characters 0 to 31 (non printable characters) The minus (hyphen) character can be used to specify a range of characters in a character class. For example, [d-m] matches any letter between d and m, inclusive. If a minus character is required in a class, it must be escaped with a backslash or appear in a position where it cannot be interpreted as indicating a range, typically as the first or last character in the class. Ranges operate in ASCII collating sequence. They can also be used for characters specified numerically, for example [\000-\037].

TextPipe also provides some convenient short cuts for commonly used classes. They may be used either on their own or inside a character class.

. | Any character, including newline (by default)
\d | any decimal digit
\D | any character that is not a decimal digit
\s | any white space character. space, formfeed, newline, carriage return, horizontal tab, and vertical tab
\S | any character that is not a white space character
\w | any "word" character. A "word" character is any letter or digit or the underscore character. The definition of letters and digits may vary, for example, in the "fr" (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.
\W | any "non-word" character

You can also describe 'special' or unprintable characters using the following patterns:

\t | tab (decimal 9, hex 09)
\f | form feed (decimal 12, hex 0C)
\n | new line (decimal 10, hex 0A)
\r | carriage return (decimal 13, hex 0D)
\a | alarm, that is, the BEL character (decimal 7, hex 07)
\e | escape (decimal 27, hex 1B)
\cx | "control-x", where x is any character The precise effect of "\cx" is as follows: if "x" is a lower case letter, it is converted to upper case. Then bit 6 of the character (hex 40) is inverted. Thus "\cz" becomes hex 1A, but "\c{" becomes hex 3B, while "\c;" becomes hex 7B.
\xhh | character with hex code hh After "\x", up to two hexadecimal digits are read (letters can be in upper or lower case).
\ddd | character with octal code ddd, or back reference After "\0" up to two further octal digits are read. In both cases, if there are fewer than two digits, just those that are present are used. Thus the sequence "\0\x\07" specifies two binary zeros followed by a BEL character. Make sure you supply two digits after the initial zero if the character that follows is itself an octal digit.

Repetition - quantifiers

Often you are looking for a pattern that is repeated. TextPipe provides a number of different methods to specify how many times a pattern is allowed to repeat. Each method is called a quantifier.

The general repetition quantifier specifies a minimum and maximum number of permitted matches, by giving the two numbers in curly brackets (braces), separated by a comma. The numbers must be less than 65536, and the first must be less than or equal to the second. For example:

z{2,4}

matches "zz", "zzz", or "zzzz". A closing brace on its own is not a special character. If the second number is omitted, but the comma is present, there is no upper limit; if the second number and the comma are both omitted, the quantifier specifies an exact number of required matches. Thus

[aeiou]{3,}

matches at least 3 successive vowels, but may match many more, while

\d{8}

matches exactly 8 digits. An opening curly bracket that appears in a position where a quantifier is not allowed, or one that does not match the syntax of a quantifier, is taken as a literal character. For example, {,6} is not a quantifier, but a literal string of four characters. For convenience (and historical compatibility) the three most common quantifiers have single-character abbreviations:

* is equivalent to {0,}

+ is equivalent to {1,}

? is equivalent to {0,1}

Normally TextPipe will try to repeat matches as few times as possible to make the rest of the pattern match - this is known as non-greedy (or minimal) matching. Others programs can use greedy or maximal matching by default. Using the Search and Replace Pattern Options dialog you can change this to greedy (or maximal) matching. If a quantifier is followed by a question mark, it inverts the default 'greediness' of the quantifier relative to what is specified in the Pattern Options dialog.

Pattern | Pattern Options Dialog set to... | Subject text (match shown in bold)
<.*> | Default: Shortest (minimal, non-greedy) match | <html><p>Hello there</p></html>
<.*> | Default: Longest (maximal, greedy) match | <html><p>Hello there</p></html>
<.*?> | Default: Shortest (minimal, non-greedy) match | <html><p>Hello there</p></html>
<.*?> | Default: Longest (maximal, greedy) match | <html><p>Hello there</p></html>

Using these quantifiers we can now build the following examples:

a+ or a+? (depending on greediness setting) | a, aa, aaa, aaaa etc
\d+ or \d+? (depending on greediness setting) | any decimal integer
\w+ or \w+? (depending on greediness setting) | any single word
\d{2,3} \d{3,4} \d{4} | A 7 or 8 digit phone number with a 2 or 3 digit area code
s.t | Matches the real words set, sat, sit (and others)

Alternatives

Vertical bar ('pipe') characters are used to separate alternative patterns. For example, the pattern

gilbert|sullivan

matches either "gilbert" or "sullivan". Any number of alternatives may appear, and an empty alternative is permitted (matching the empty string). The matching process tries each alternative in turn, from left to right, and the first one that succeeds is used. If the alternatives are within a sub pattern (defined below), "succeeds" means matching the rest of the main pattern as well as the alternative in the sub pattern.

Positional matching or anchors

Sometime you only want a pattern to match when it is found at the start of a file, or the end of a line, or when it's a whole word. Various positional operators force the match to only occur when it's found in a specific position.

^

Almost always used at the start of a pattern, the circumflex forces the match to only occur at the start of a line (i.e. after a \n or ASCII 10) or at the start of the file.

Circumflex need not be the first character of the pattern if a number of alternatives are involved, but it should be the first thing in each alternative in which it appears if the pattern is ever to match that branch.

Instead of Circumflex, consider using the following assertion:

(?<=\r|\n|\A)

The reason for this change is a little technical. TextPipe's pattern matching engine expects Unix end of line characters (\n or ASCII 10), so Circumflex will work for Unix or PC files, but fails utterly on Macintosh files unless you use the assertion above or explicitly convert the file first.

$

Almost always used at the end of a pattern, the dollar character forces the match to only occur at the end of a line (i.e. before a \n or ASCII 10), or at the end of file.

Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

Instead of Dollar, consider using the following assertion:

(?=\r|\n|\z)

The reason for this change is a little technical. TextPipe's pattern matching engine expects Unix end of line characters (\n or ASCII 10), so Dollar will work on Unix files, fail utterly on Macintosh files, and on PC files it will include the preceding \r (ASCII 13) as part of the line unless you use the assertion above or explicitly convert the file first.

\b

word boundary

A word boundary is a position in the subject string where the current character and the previous character do not both match \w or \W (i.e. one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively.

\B

not a word boundary

\A

start of file

\Z

end of file or new line at end of file

\z

end of file

The difference between \Z and \z is that \Z matches before a new line that is the last character of the string as well as at the end of the string, whereas \z matches only at the end.

Sub Patterns and the Replacement String

Sub patterns are delimited by parentheses (round brackets), which can be nested. Marking part of a pattern as a sub pattern does two things:

1. It localizes a set of alternatives. For example, the pattern

cat(aract|erpillar|)

matches one of the words "cat", "cataract", or "caterpillar". Without the parentheses, it would match "cataract", "erpillar" or the empty string.

2. It sets up the sub pattern as a capturing sub pattern. The text matches by a capturing sub pattern can be referred to later (such as in the replacement string) using the macros $0 (for the full matching text), $1 (for the first sub pattern), $2 ... $9, $a ...$z etc. Opening parentheses are counted from left to right (starting from 1) to obtain the numbers of the capturing sub patterns.

For example, if the string "the red king" is matched against the pattern

the ((red|white) (king|queen))

the captured substrings are "red king", "red", and "king", and are numbered 1, 2, and 3.

The fact that plain parentheses fulfill two functions is not always helpful. There are often times when a grouping sub pattern is required without a capturing requirement. If an opening parenthesis is followed by "?:", the sub pattern does not do any capturing, and is not counted when computing the number of any subsequent capturing sub patterns. For example, if the string "the white queen" is matched against the pattern

the ((?:red|white) (king|queen))

the captured sub strings are "white queen" and "queen", and are numbered 1 and 2. The maximum number of captured sub strings is 36 (0-9, a-z), and the maximum number of all sub patterns, both capturing and non-capturing, is 200.

Using parentheses appropriately in the search expression lets the program remember found text, to be used as replacement text. The simplest example of this, without the need for parentheses, is the complete found string, represented by the ‘$0’ character in the replacement string. For example, if your search Regular Expression was ‘test|trial|experiment’, and your replacement string was ‘<b>$0</b>’, every instance of the word ‘test’ in your document would be replaced by ‘<b>test</b>’, and similarly for ‘trial’ and ‘experiment’ (assuming this was an HTML document, this would have the effect of bolding these words). Note that if you want to include an actual ‘$’ in your replacement text, escape it, as in, ‘$$’.

You can use parentheses to remember specific parts of the found text. For example, the Regular Expression ‘<img src="([^"]+)">’ would find and match image tags in an HTML document (assuming they were formed exactly like this), and would remember the source image file. You can recall this by using ‘$1’ in your replacement string. So if your replacement string was ‘-- Image File: $1 --’, and the HTML file processed contained the string ‘<img src="/images/test.gif">’, that string would be replaced by ‘-- Image File: /images/test.gif --’.

You can remember multiple parts of the found text. For example, the Regular Expression ‘<img src="([^"]+)" alt="([^"]+)">’ would find and match a string such as ‘<img src="/images/test.gif" alt="My Cool Image">’. If the replacement string was ‘$2 ($1)’, then this image would be replaced by ‘My Cool Image (/images/test.gif)’.

Performance

The time which a search for a Regular Expressions takes can range from unimportant to unbearably slow. Seemingly small changes in a Regular Expression can make a world of difference. For example, the regular expression

‘[^"]*("[^"]+")[^"]*’

finds and matches quoted strings (which it remembers), as well as surrounding text without quotes. This particular Regular Expression executes quickly enough (counted in seconds, or less) if the file being processed actually has quoted strings in it. If, however, the file is of a reasonable size (say, 50 KB), and does not have any quoted strings in it, the search for this regular expression will take an incredible amount of time to complete – maybe minutes! It turns out that it is the first piece of the expression that is causing the problem. Removing it makes the search on the 50 KB file nearly instantaneous. Why? Because now the search is smart enough to realize that before it bothers matching anything else, it must at least find a double-quote. When it doesn’t find one, it’s finished it’s search.

So one trick to making fast regular expressions, is to form the regular expression in such a way that it can fail as early as possible. Try to put strings that must be matched, right at the beginning of the regular expression. Apart from this, experiment and see what works well and what doesn’t.

Things To Note - Escaping Meta characters

If you want to find any of the meta characters on their own, you must escape them with a backslash to prevent them being interpreted. When in doubt, quote all non-alphanumeric characters. The meta characters are:

Meta character | Purpose
\ | first character of assertions, such as \w or \d
| | OR; allows matching options
. | match any character
( ... ) | grouping operator; builds $1, $2, etc.
[ ... ] | match any character within brackets
+ | quantifier: match one or more times
? | quantifier: at most one match
* | quantifier: match zero or more times
{n,m} | quantifier: match between n and m times
^ | match at start of line
$ | match at end of line

Further reading

For more information on performance, assertions, back references, limitations, POSIX character classes and more details on the above text, please see the Perl/PCRE pattern matching reference.


Pattern Matching Examples

 Pattern Matching Examples
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The pattern below are all for Perl-style patterns.

Some Basic Regular Expression Search Examples

What to Match | Operator | Examples
Any single character | . | g.t matches get, got, gut, together, g-t
Any string of characters (one or more) | + | w+e matches were, wwezaa, wwwe.write (longest possible match is used)
Any string of characters (or none) | * | w*e matches were, were, wwezaa, wwwe.write (longest possible match is used)
One of the specified characters | [] | g[eo]t matches get and got but not gut
One of the characters in a range | [-] | [b-p]at matches bat, cat, fat, hat, mat but not rat or sat
One expression or another | (|) | W(in|indows) matches Win or Windows
One or more expressions | ()+ | (at)+ matches atat in catatonic and at in battle
All characters (perhaps on different lines) | .* | h.*d matches helped, Hello World, & Hello (cr lf) Win95 World.
Two strings “nearby” (perhaps on different lines) | .* | the[\0-ÿ]*first matches the first and the (cr lf) very first
One of the characters not in a range | [^-] | [^b-p]at matches at in rat & sat and nothing in bat, cat, fat, hat.
Find all words of length x |  | [^\w]\w\w\w+ will find all 3 letter words, [^\w]a\w\w\w\w finds all 5 letter words starting with 'a'.

Additional Regular Expression Search Examples

Search String | Match
L. | line, list, lane, LogVar1 (when case sensitive is off)
[on][on] | non, no, on
\[ | [
[a-z]+ | abcdefghijklomopqrstuvwxyz Note: If used with Case Sensitive off, it will also match ABCDEFGHIJKLMNOPQRSTUVWXYZ
[a-zA-ZÀ-ÿ] | Matches the same as the +[a-z] example above (regardless of case sensitive flag) and also all other special characters for Western European languages.
(one|two|three) | one, two, three
(one|two|three)+ | one, onetwo, twothree
Windows[\0-ÿ]*95 | Will match up to 4096 characters (on several lines) between Windows and 95.
All .* replaces | all the replaces (the .* character matches all chars)
/\*[\0-ÿ]*\*/ | Matches C style comments on several lines if necessary (up to 4096 characters)
01[0-9]*.htm | 01346.htm, 01.htm, 016965.htm
[ab]*c | c, abc, bac, abbc, bbac
.include (<|\[)+[a-z0-9_].h*(p)+[\]>] | Matches: #include [stdafx.h] #include <dos.h> #include [my_include.hpp] #include [sr32.h]

See also

Regular expressions


Pattern Matching Reference

 Pattern Matching Reference
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe's Perl compatible regular expressions are provided by the PCRE library, described below in detail.

Note: The special character '$' (end of line) may not work as expected because PCRE expects Unix end of lines. Use the expression [\r\n] instead.

· PCRE REGULAR EXPRESSION DETAILS
· BACKSLASH
· CIRCUMFLEX AND DOLLAR
· FULL STOP (PERIOD, DOT)
· MATCHING A SINGLE BYTE
· SQUARE BRACKETS
· POSIX CHARACTER CLASSES
· VERTICAL BAR
· INTERNAL OPTION SETTING
· SUBPATTERNS
· NAMED SUBPATTERNS
· REPETITION
· ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS
· BACK REFERENCES
· ASSERTIONS
· CONDITIONAL SUBPATTERNS
· COMMENTS
· RECURSIVE PATTERNS - not supported
· CALLOUTS - not supported
· UTF-8 Support
· Differences from Perl

PCRE REGULAR EXPRESSION DETAILS

The syntax and semantics of the regular expressions supported by PCRE are described below. Regular expressions are also described in the Perl documentation and in a number of other books, some of which have copious examples. Jeffrey Friedl's "Mastering Regular Expressions", published by O'Reilly, covers them in great detail. The description here is intended as reference documentation.

The basic operation of PCRE is on strings of bytes. However, there is also support for UTF-8 character strings. To use this support you must build PCRE to include UTF-8 support, and then call pcre_compile() with the PCRE_UTF8 option. How this affects the pattern matching is mentioned in several places below.

A regular expression is a pattern that is matched against a subject string from left to right. Most characters stand for themselves in a pattern, and match the corresponding characters in the subject. As a trivial example, the pattern

  The quick brown fox

matches a portion of a subject string that is identical to itself. The power of regular expressions comes from the ability to include alternatives and repetitions in the pattern. These are encoded in the pattern by the use of meta-characters, which do not stand for themselves but instead are interpreted in some special way.

There are two different sets of meta-characters: those that are recognized anywhere in the pattern except within square brackets, and those that are recognized in square brackets. Outside square brackets, the meta-characters are as follows:

  \      general escape character with several uses

  ^      assert start of string (or line, in multiline mode)

  $      assert end of string (or line, in multiline mode)

  .      match any character except newline (by default)

  [      start character class definition

  |      start of alternative branch

  (      start subpattern

  )      end subpattern

  ?      extends the meaning of (

         also 0 or 1 quantifier

         also quantifier minimizer

  *      0 or more quantifier

  +      1 or more quantifier

         also "possessive quantifier"

  {      start min/max quantifier

Part of a pattern that is in square brackets is called a "character class". In a character class the only meta-characters are:

  \      general escape character

  ^      negate the class, but only if the first character

  -      indicates character range

  [      POSIX character class (only if followed by POSIX

           syntax)

  ]      terminates the character class

The following sections describe the use of each of the meta-characters.

BACKSLASH

The backslash character has several uses. Firstly, if it is followed by a non-alphanumeric character, it takes away any special meaning that character may have. This use of backslash as an escape character applies both inside and outside character classes.

For example, if you want to match a * character, you write \* in the pattern. This escaping action applies whether or not the following character would otherwise be interpreted as a meta-character, so it is always safe to precede a non-alphameric with backslash to specify that it stands for itself. In particular, if you want to match a backslash, you write \\.

If a pattern is compiled with the PCRE_EXTENDED option, whitespace in the pattern (other than in a character class) and characters between a # outside a character class and the next newline character are ignored. An escaping backslash can be used to include a whitespace or # character as part of the pattern.

If you want to remove the special meaning from a sequence of characters, you can do so by putting them between \Q and \E. This is different from Perl in that $ and @ are handled as literals in \Q...\E sequences in PCRE, whereas in Perl, $ and @ cause variable interpolation. Note the following examples:

  Pattern            PCRE matches   Perl matches

\Qabc$xyz\E        abc$xyz        abc followed by the contents of $xyz

  \Qabc\$xyz\E       abc\$xyz       abc\$xyz

  \Qabc\E\$\Qxyz\E   abc$xyz        abc$xyz

The \Q...\E sequence is recognized both inside and outside character classes.

A second use of backslash provides a way of encoding non-printing characters in patterns in a visible manner. There is no restriction on the appearance of non-printing characters, apart from the binary zero that terminates a pattern, but when a pattern is being prepared by text editing, it is usually easier to use one of the following escape sequences than the binary character it represents:

  \a        alarm, that is, the BEL character (hex 07)

  \cx       "control-x", where x is any character

  \e        escape (hex 1B)

  \f        formfeed (hex 0C)

  \n        newline (hex 0A)

  \r        carriage return (hex 0D)

  \t        tab (hex 09)

  \ddd      character with octal code ddd, or backreference

  \xhh      character with hex code hh

  \x{hhh..} character with hex code hhh... (UTF-8 mode only)

The precise effect of \cx is as follows: if x is a lower case letter, it is converted to upper case. Then bit 6 of the character (hex 40) is inverted. Thus \cz becomes hex 1A, but \c{ becomes hex 3B, while \c; becomes hex 7B.

After \x, from zero to two hexadecimal digits are read (letters can be in upper or lower case). In UTF-8 mode, any number of hexadecimal digits may appear between \x{ and }, but the value of the character code must be less than 2**31 (that is, the maximum hexadecimal value is 7FFFFFFF). If characters other than hexadecimal digits appear between \x{ and }, or if there is no terminating }, this form of escape is not recognized. Instead, the initial \x will be interpreted as a basic hexadecimal escape, with no following digits, giving a byte whose value is zero.

Characters whose value is less than 256 can be defined by either of the two syntaxes for \x when PCRE is in UTF-8 mode. There is no difference in the way they are handled. For example, \xdc is exactly the same as \x{dc}.

After \0 up to two further octal digits are read. In both cases, if there are fewer than two digits, just those that are present are used. Thus the sequence \0\x\07 specifies two binary zeros followed by a BEL character (code value 7). Make sure you supply two digits after the initial zero if the character that follows is itself an octal digit.

The handling of a backslash followed by a digit other than 0 is complicated. Outside a character class, PCRE reads it and any following digits as a decimal number. If the number is less than 10, or if there have been at least that many previous capturing left parentheses in the expression, the entire sequence is taken as a back reference. A description of how this works is given later, following the discussion of parenthesized subpatterns.

Inside a character class, or if the decimal number is greater than 9 and there have not been that many capturing subpatterns, PCRE re-reads up to three octal digits following the backslash, and generates a single byte from the least significant 8 bits of the value. Any subsequent digits stand for themselves. For example:

  \040   is another way of writing a space

  \40    is the same, provided there are fewer than 40

            previous capturing subpatterns

  \7     is always a back reference

  \11    might be a back reference, or another way of

            writing a tab

  \011   is always a tab

  \0113  is a tab followed by the character "3"

  \113   might be a back reference, otherwise the

            character with octal code 113

  \377   might be a back reference, otherwise

            the byte consisting entirely of 1 bits

  \81    is either a back reference, or a binary zero

            followed by the two characters "8" and "1"

Note that octal values of 100 or greater must not be introduced by a leading zero, because no more than three octal digits are ever read.

All the sequences that define a single byte value or a single UTF-8 character (in UTF-8 mode) can be used both inside and outside character classes. In addition, inside a character class, the sequence \b is interpreted as the backspace character (hex 08). Outside a character class it has a different meaning (see below).

The third use of backslash is for specifying generic character types:

  \d     any decimal digit

  \D     any character that is not a decimal digit

  \s     any whitespace character

  \S     any character that is not a whitespace character

  \w     any "word" character

  \W     any "non-word" character

Each pair of escape sequences partitions the complete set of characters into two disjoint sets. Any given character matches one, and only one, of each pair.

In UTF-8 mode, characters with values greater than 255 never match \d, \s, or \w, and always match \D, \S, and \W.

For compatibility with Perl, \s does not match the VT character (code 11). This makes it different from the the POSIX "space" class. The \s characters are HT (9), LF (10), FF (12), CR (13), and space (32).

A "word" character is any letter or digit or the underscore character, that is, any character which can be part of a Perl "word". The definition of letters and digits is controlled by PCRE's character tables, and may vary if locale- specific matching is taking place (see "Locale support" in the pcreapi page). For example, in the "fr" (French) locale, some character codes greater than 128 are used for accented letters, and these are matched by \w.

These character type sequences can appear both inside and outside character classes. They each match one character of the appropriate type. If the current matching point is at the end of the subject string, all of them fail, since there is no character to match.

The fourth use of backslash is for certain simple assertions. An assertion specifies a condition that has to be met at a particular point in a match, without consuming any characters from the subject string. The use of subpatterns for more complicated assertions is described below. The backslashed assertions are

  \b     matches at a word boundary

  \B     matches when not at a word boundary

  \A     matches at start of subject

  \Z     matches at end of subject or before newline at end

  \z     matches at end of subject

  \G     matches at first matching position in subject

These assertions may not appear in character classes (but note that \b has a different meaning, namely the backspace character, inside a character class).

A word boundary is a position in the subject string where the current character and the previous character do not both match \w or \W (i.e. one matches \w and the other matches \W), or the start or end of the string if the first or last character matches \w, respectively.

The \A, \Z, and \z assertions differ from the traditional circumflex and dollar (described below) in that they only ever match at the very start and end of the subject string, whatever options are set. Thus, they are independent of multiline mode.

They are not affected by the PCRE_NOTBOL or PCRE_NOTEOL options. If the startoffset argument of pcre_exec() is non-zero, indicating that matching is to start at a point other than the beginning of the subject, \A can never match. The difference between \Z and \z is that \Z matches before a newline that is the last character of the string as well as at the end of the string, whereas \z matches only at the end.

The \G assertion is true only when the current matching position is at the start point of the match, as specified by the startoffset argument of pcre_exec(). It differs from \A when the value of startoffset is non-zero. By calling pcre_exec() multiple times with appropriate arguments, you can mimic Perl's /g option, and it is in this kind of implementation where \G can be useful.

Note, however, that PCRE's interpretation of \G, as the start of the current match, is subtly different from Perl's, which defines it as the end of the previous match. In Perl, these can be different when the previously matched string was empty. Because PCRE does just one match at a time, it cannot reproduce this behaviour.

If all the alternatives of a pattern begin with \G, the expression is anchored to the starting match position, and the "anchored" flag is set in the compiled regular expression.

CIRCUMFLEX AND DOLLAR

Outside a character class, in the default matching mode, the circumflex character is an assertion which is true only if the current matching point is at the start of the subject string. If the startoffset argument of pcre_exec() is non-zero, circumflex can never match if the PCRE_MULTILINE option is unset. Inside a character class, circumflex has an entirely different meaning (see below).

Circumflex need not be the first character of the pattern if a number of alternatives are involved, but it should be the first thing in each alternative in which it appears if the pattern is ever to match that branch. If all possible alternatives start with a circumflex, that is, if the pattern is constrained to match only at the start of the subject, it is said to be an "anchored" pattern. (There are also other constructs that can cause a pattern to be anchored.)

A dollar character is an assertion which is true only if the current matching point is at the end of the subject string, or immediately before a newline character that is the last character in the string (by default). Dollar need not be the last character of the pattern if a number of alternatives are involved, but it should be the last item in any branch in which it appears. Dollar has no special meaning in a character class.

The meaning of dollar can be changed so that it matches only at the very end of the string, by setting the PCRE_DOLLAR_ENDONLY option at compile time. This does not affect the \Z assertion.

The meanings of the circumflex and dollar characters are changed if the PCRE_MULTILINE option is set. When this is the case, they match immediately after and immediately before an internal newline character, respectively, in addition to matching at the start and end of the subject string. For example, the pattern /^abc$/ matches the subject string "def\nabc" in multiline mode, but not otherwise. Consequently, patterns that are anchored in single line mode because all branches start with ^ are not anchored in multiline mode, and a match for circumflex is possible when the startoffset argument of pcre_exec() is non-zero. The PCRE_DOLLAR_ENDONLY option is ignored if PCRE_MULTILINE is set.

Note that the sequences \A, \Z, and \z can be used to match the start and end of the subject in both modes, and if all branches of a pattern start with \A it is always anchored, whether PCRE_MULTILINE is set or not.

FULL STOP (PERIOD, DOT)

Outside a character class, a dot in the pattern matches any one character in the subject, including a non-printing character, but not (by default) newline. In UTF-8 mode, a dot matches any UTF-8 character, which might be more than one byte long, except (by default) for newline. If the PCRE_DOTALL option is set, dots match newlines as well. The handling of dot is entirely independent of the handling of circumflex and dollar, the only relationship being that they both involve newline characters. Dot has no special meaning in a character class.

MATCHING A SINGLE BYTE

Outside a character class, the escape sequence \C matches any one byte, both in and out of UTF-8 mode. Unlike a dot, it always matches a newline. The feature is provided in Perl in order to match individual bytes in UTF-8 mode. Because it breaks up UTF-8 characters into individual bytes, what remains in the string may be a malformed UTF-8 string. For this reason it is best avoided.

PCRE does not allow \C to appear in lookbehind assertions (see below), because in UTF-8 mode it makes it impossible to calculate the length of the lookbehind.

SQUARE BRACKETS

An opening square bracket introduces a character class, terminated by a closing square bracket. A closing square bracket on its own is not special. If a closing square bracket is required as a member of the class, it should be the first data character in the class (after an initial circumflex, if present) or escaped with a backslash.

A character class matches a single character in the subject. In UTF-8 mode, the character may occupy more than one byte. A matched character must be in the set of characters defined by the class, unless the first character in the class definition is a circumflex, in which case the subject character must not be in the set defined by the class. If a circumflex is actually required as a member of the class, ensure it is not the first character, or escape it with a backslash.

For example, the character class [aeiou] matches any lower case vowel, while [^aeiou] matches any character that is not a lower case vowel. Note that a circumflex is just a convenient notation for specifying the characters which are in the class by enumerating those that are not. It is not an assertion: it still consumes a character from the subject string, and fails if the current pointer is at the end of the string.

In UTF-8 mode, characters with values greater than 255 can be included in a class as a literal string of bytes, or by using the \x{ escaping mechanism.

When caseless matching is set, any letters in a class represent both their upper case and lower case versions, so for example, a caseless [aeiou] matches "A" as well as "a", and a caseless [^aeiou] does not match "A", whereas a caseful version would. PCRE does not support the concept of case for characters with values greater than 255.

The newline character is never treated in any special way in character classes, whatever the setting of the PCRE_DOTALL or PCRE_MULTILINE options is. A class such as [^a] will always match a newline.

The minus (hyphen) character can be used to specify a range of characters in a character class. For example, [d-m] matches any letter between d and m, inclusive. If a minus character is required in a class, it must be escaped with a backslash or appear in a position where it cannot be interpreted as indicating a range, typically as the first or last character in the class.

It is not possible to have the literal character "]" as the end character of a range. A pattern such as [W-]46] is interpreted as a class of two characters ("W" and "-") followed by a literal string "46]", so it would match "W46]" or "-46]". However, if the "]" is escaped with a backslash it is interpreted as the end of range, so [W-\]46] is interpreted as a single class containing a range followed by two separate characters. The octal or hexadecimal representation of "]" can also be used to end a range.

Ranges operate in the collating sequence of character values. They can also be used for characters specified numerically, for example [\000-\037]. In UTF-8 mode, ranges can include characters whose values are greater than 255, for example [\x{100}-\x{2ff}].

If a range that includes letters is used when caseless matching is set, it matches the letters in either case. For example, [W-c] is equivalent to [][\^_`wxyzabc], matched caselessly, and if character tables for the "fr" locale are in use, [\xc8-\xcb] matches accented E characters in both cases.

The character types \d, \D, \s, \S, \w, and \W may also appear in a character class, and add the characters that they match to the class. For example, [\dABCDEF] matches any hexadecimal digit. A circumflex can conveniently be used with the upper case character types to specify a more restricted set of characters than the matching lower case type. For example, the class [^\W_] matches any letter or digit, but not underscore.

All non-alphameric characters other than \, -, ^ (at the start) and the terminating ] are non-special in character classes, but it does no harm if they are escaped.

POSIX CHARACTER CLASSES

Perl supports the POSIX notation for character classes, which uses names enclosed by [: and :] within the enclosing square brackets. PCRE also supports this notation. For example,

  [01[:alpha:]%]

matches "0", "1", any alphabetic character, or "%". The supported class names are

  alnum    letters and digits

  alpha    letters

  ascii    character codes 0 - 127

  blank    space or tab only

  cntrl    control characters

  digit    decimal digits (same as \d)

  graph    printing characters, excluding space

  lower    lower case letters

  print    printing characters, including space

  punct    printing characters, excluding letters and digits

  space    white space (not quite the same as \s)

  upper    upper case letters

  word     "word" characters (same as \w)

  xdigit   hexadecimal digits

The "space" characters are HT (9), LF (10), VT (11), FF (12), CR (13), and space (32). Notice that this list includes the VT character (code 11). This makes "space" different to \s, which does not include VT (for Perl compatibility).

The name "word" is a Perl extension, and "blank" is a GNU extension from Perl 5.8. Another Perl extension is negation, which is indicated by a ^ character after the colon. For example,

  [12[:^digit:]]

matches "1", "2", or any non-digit. PCRE (and Perl) also recognize the POSIX syntax [.ch.] and [=ch=] where "ch" is a "collating element", but these are not supported, and an error is given if they are encountered.

In UTF-8 mode, characters with values greater than 255 do not match any of the POSIX character classes.

VERTICAL BAR

Vertical bar characters are used to separate alternative patterns. For example, the pattern

  gilbert|sullivan

matches either "gilbert" or "sullivan". Any number of alternatives may appear, and an empty alternative is permitted (matching the empty string). The matching process tries each alternative in turn, from left to right, and the first one that succeeds is used. If the alternatives are within a subpattern (defined below), "succeeds" means matching the rest of the main pattern as well as the alternative in the subpattern.

INTERNAL OPTION SETTING

The settings of the PCRE_CASELESS, PCRE_MULTILINE, PCRE_DOTALL, and PCRE_EXTENDED options can be changed from within the pattern by a sequence of Perl option letters enclosed between "(?" and ")". The option letters are

  i  for PCRE_CASELESS

  m  for PCRE_MULTILINE

  s  for PCRE_DOTALL

  x  for PCRE_EXTENDED

For example, (?im) sets caseless, multiline matching. It is also possible to unset these options by preceding the letter with a hyphen, and a combined setting and unsetting such as (?im-sx), which sets PCRE_CASELESS and PCRE_MULTILINE while unsetting PCRE_DOTALL and PCRE_EXTENDED, is also permitted. If a letter appears both before and after the hyphen, the option is unset.

When an option change occurs at top level (that is, not inside subpattern parentheses), the change applies to the remainder of the pattern that follows. If the change is placed right at the start of a pattern, PCRE extracts it into the global options (and it will therefore show up in data extracted by the pcre_fullinfo() function).

An option change within a subpattern affects only that part of the current pattern that follows it, so

  (a(?i)b)c

matches abc and aBc and no other strings (assuming PCRE_CASELESS is not used). By this means, options can be made to have different settings in different parts of the pattern. Any changes made in one alternative do carry on into subsequent branches within the same subpattern. For example,

  (a(?i)b|c)

matches "ab", "aB", "c", and "C", even though when matching "C" the first branch is abandoned before the option setting. This is because the effects of option settings happen at compile time. There would be some very weird behaviour otherwise.

The PCRE-specific options PCRE_UNGREEDY and PCRE_EXTRA can be changed in the same way as the Perl-compatible options by using the characters U and X respectively. The (?X) flag setting is special in that it must always occur earlier in the pattern than any of the additional features it turns on, even when it is at top level. It is best put at the start.

SUBPATTERNS

Subpatterns are delimited by parentheses (round brackets), which can be nested. Marking part of a pattern as a subpattern does two things:

1. It localizes a set of alternatives. For example, the pattern

  cat(aract|erpillar|)

matches one of the words "cat", "cataract", or "caterpillar". Without the parentheses, it would match "cataract", "erpillar" or the empty string.

2. It sets up the subpattern as a capturing subpattern (as defined above). When the whole pattern matches, that portion of the subject string that matched the subpattern is passed back to the caller via the ovector argument of pcre_exec(). Opening parentheses are counted from left to right (starting from 1) to obtain the numbers of the capturing subpatterns.

For example, if the string "the red king" is matched against the pattern

  the ((red|white) (king|queen))

the captured substrings are "red king", "red", and "king", and are numbered 1, 2, and 3, respectively.

The fact that plain parentheses fulfil two functions is not always helpful. There are often times when a grouping subpattern is required without a capturing requirement. If an opening parenthesis is followed by a question mark and a colon, the subpattern does not do any capturing, and is not counted when computing the number of any subsequent capturing subpatterns. For example, if the string "the white queen" is matched against the pattern

  the ((?:red|white) (king|queen))

the captured substrings are "white queen" and "queen", and are numbered 1 and 2. The maximum number of capturing subpatterns is 65535, and the maximum depth of nesting of all subpatterns, both capturing and non-capturing, is 200.

As a convenient shorthand, if any option settings are required at the start of a non-capturing subpattern, the option letters may appear between the "?" and the ":". Thus the two patterns

  (?i:saturday|sunday)

  (?:(?i)saturday|sunday)

match exactly the same set of strings. Because alternative branches are tried from left to right, and options are not reset until the end of the subpattern is reached, an option setting in one branch does affect subsequent branches, so the above patterns match "SUNDAY" as well as "Saturday".

NAMED SUBPATTERNS

Identifying capturing parentheses by number is simple, but it can be very hard to keep track of the numbers in complicated regular expressions. Furthermore, if an expression is modified, the numbers may change. To help with the difficulty, PCRE supports the naming of subpatterns, something that Perl does not provide. The Python syntax (?P<name>...) is used. Names consist of alphanumeric characters and underscores, and must be unique within a pattern.

Named capturing parentheses are still allocated numbers as well as names. The PCRE API provides function calls for extracting the name-to-number translation table from a compiled pattern. For further details see the pcreapi documentation.

REPETITION

Repetition is specified by quantifiers, which can follow any of the following items:

  a literal data character

  the . metacharacter

  the \C escape sequence

  escapes such as \d that match single characters

  a character class

  a back reference (see next section)

  a parenthesized subpattern (unless it is an assertion)

The general repetition quantifier specifies a minimum and maximum number of permitted matches, by giving the two numbers in curly brackets (braces), separated by a comma. The numbers must be less than 65536, and the first must be less than or equal to the second. For example:

  z{2,4}

matches "zz", "zzz", or "zzzz". A closing brace on its own is not a special character. If the second number is omitted, but the comma is present, there is no upper limit; if the second number and the comma are both omitted, the quantifier specifies an exact number of required matches. Thus

  [aeiou]{3,}

matches at least 3 successive vowels, but may match many more, while

  \d{8}

matches exactly 8 digits. An opening curly bracket that appears in a position where a quantifier is not allowed, or one that does not match the syntax of a quantifier, is taken as a literal character. For example, {,6} is not a quantifier, but a literal string of four characters.

In UTF-8 mode, quantifiers apply to UTF-8 characters rather than to individual bytes. Thus, for example, \x{100}{2} matches two UTF-8 characters, each of which is represented by a two-byte sequence.

The quantifier {0} is permitted, causing the expression to behave as if the previous item and the quantifier were not present.

For convenience (and historical compatibility) the three most common quantifiers have single-character abbreviations:

  *    is equivalent to {0,}

  +    is equivalent to {1,}

  ?    is equivalent to {0,1}

It is possible to construct infinite loops by following a subpattern that can match no characters with a quantifier that has no upper limit, for example:

  (a?)*

Earlier versions of Perl and PCRE used to give an error at compile time for such patterns. However, because there are cases where this can be useful, such patterns are now accepted, but if any repetition of the subpattern does in fact match no characters, the loop is forcibly broken.

By default, the quantifiers are "greedy", that is, they match as much as possible (up to the maximum number of permitted times), without causing the rest of the pattern to fail. The classic example of where this gives problems is in trying to match comments in C programs. These appear between the sequences /* and */ and within the sequence, individual * and / characters may appear. An attempt to match C comments by applying the pattern

  /\*.*\*/

to the string

  /* first command */  not comment  /* second comment */

fails, because it matches the entire string owing to the greediness of the .* item.

However, if a quantifier is followed by a question mark, it ceases to be greedy, and instead matches the minimum number of times possible, so the pattern

  /\*.*?\*/

does the right thing with the C comments. The meaning of the various quantifiers is not otherwise changed, just the preferred number of matches. Do not confuse this use of question mark with its use as a quantifier in its own right. Because it has two uses, it can sometimes appear doubled, as in

  \d??\d

which matches one digit by preference, but can match two if that is the only way the rest of the pattern matches.

If the PCRE_UNGREEDY option is set (an option which is not available in Perl), the quantifiers are not greedy by default, but individual ones can be made greedy by following them with a question mark. In other words, it inverts the default behaviour.

When a parenthesized subpattern is quantified with a minimum repeat count that is greater than 1 or with a limited maximum, more store is required for the compiled pattern, in proportion to the size of the minimum or maximum.

If a pattern starts with .* or .{0,} and the PCRE_DOTALL option (equivalent to Perl's /s) is set, thus allowing the . to match newlines, the pattern is implicitly anchored, because whatever follows will be tried against every character position in the subject string, so there is no point in retrying the overall match at any position after the first. PCRE normally treats such a pattern as though it were preceded by \A.

In cases where it is known that the subject string contains no newlines, it is worth setting PCRE_DOTALL in order to obtain this optimization, or alternatively using ^ to indicate anchoring explicitly.

However, there is one situation where the optimization cannot be used. When .* is inside capturing parentheses that are the subject of a backreference elsewhere in the pattern, a match at the start may fail, and a later one succeed. Consider, for example:

  (.*)abc\1

If the subject is "xyz123abc123" the match point is the fourth character. For this reason, such a pattern is not implicitly anchored.

When a capturing subpattern is repeated, the value captured is the substring that matched the final iteration. For example, after

  (tweedle[dume]{3}\s*)+

has matched "tweedledum tweedledee" the value of the captured substring is "tweedledee". However, if there are nested capturing subpatterns, the corresponding captured values may have been set in previous iterations. For example, after

  /(a|(b))+/

matches "aba" the value of the second captured substring is "b".

ATOMIC GROUPING AND POSSESSIVE QUANTIFIERS

With both maximizing and minimizing repetition, failure of what follows normally causes the repeated item to be re-evaluated to see if a different number of repeats allows the rest of the pattern to match. Sometimes it is useful to prevent this, either to change the nature of the match, or to cause it fail earlier than it otherwise might, when the author of the pattern knows there is no point in carrying on.

Consider, for example, the pattern \d+foo when applied to the subject line

  123456bar

After matching all 6 digits and then failing to match "foo", the normal action of the matcher is to try again with only 5 digits matching the \d+ item, and then with 4, and so on, before ultimately failing. "Atomic grouping" (a term taken from Jeffrey Friedl's book) provides the means for specifying that once a subpattern has matched, it is not to be re-evaluated in this way.

If we use atomic grouping for the previous example, the matcher would give up immediately on failing to match "foo" the first time. The notation is a kind of special parenthesis, starting with (?> as in this example:

  (?>\d+)bar

This kind of parenthesis "locks up" the part of the pattern it contains once it has matched, and a failure further into the pattern is prevented from backtracking into it. Backtracking past it to previous items, however, works as normal.

An alternative description is that a subpattern of this type matches the string of characters that an identical standalone pattern would match, if anchored at the current point in the subject string.

Atomic grouping subpatterns are not capturing subpatterns. Simple cases such as the above example can be thought of as a maximizing repeat that must swallow everything it can. So, while both \d+ and \d+? are prepared to adjust the number of digits they match in order to make the rest of the pattern match, (?>\d+) can only match an entire sequence of digits.

Atomic groups in general can of course contain arbitrarily complicated subpatterns, and can be nested. However, when the subpattern for an atomic group is just a single repeated item, as in the example above, a simpler notation, called a "possessive quantifier" can be used. This consists of an additional + character following a quantifier. Using this notation, the previous example can be rewritten as

  \d++bar

Possessive quantifiers are always greedy; the setting of the PCRE_UNGREEDY option is ignored. They are a convenient notation for the simpler forms of atomic group. However, there is no difference in the meaning or processing of a possessive quantifier and the equivalent atomic group.

The possessive quantifier syntax is an extension to the Perl syntax. It originates in Sun's Java package.

When a pattern contains an unlimited repeat inside a subpattern that can itself be repeated an unlimited number of times, the use of an atomic group is the only way to avoid some failing matches taking a very long time indeed. The pattern

  (\D+|<\d+>)*[!?]

matches an unlimited number of substrings that either consist of non-digits, or digits enclosed in <>, followed by either ! or ?. When it matches, it runs quickly. However, if it is applied to

  aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

it takes a long time before reporting failure. This is because the string can be divided between the two repeats in a large number of ways, and all have to be tried. (The example used [!?] rather than a single character at the end, because both PCRE and Perl have an optimization that allows for fast failure when a single character is used. They remember the last single character that is required for a match, and fail early if it is not present in the string.) If the pattern is changed to

  ((?>\D+)|<\d+>)*[!?]

sequences of non-digits cannot be broken, and failure happens quickly.

BACK REFERENCES

Outside a character class, a backslash followed by a digit greater than 0 (and possibly further digits) is a back reference to a capturing subpattern earlier (that is, to its left) in the pattern, provided there have been that many previous capturing left parentheses.

However, if the decimal number following the backslash is less than 10, it is always taken as a back reference, and causes an error only if there are not that many capturing left parentheses in the entire pattern. In other words, the parentheses that are referenced need not be to the left of the reference for numbers less than 10. See the section entitled "Backslash" above for further details of the handling of digits following a backslash.

A back reference matches whatever actually matched the capturing subpattern in the current subject string, rather than anything matching the subpattern itself (see "Subpatterns as subroutines" below for a way of doing that). So the pattern

  (sens|respons)e and \1ibility

matches "sense and sensibility" and "response and responsibility", but not "sense and responsibility". If caseful matching is in force at the time of the back reference, the case of letters is relevant. For example,

  ((?i)rah)\s+\1

matches "rah rah" and "RAH RAH", but not "RAH rah", even though the original capturing subpattern is matched caselessly.

Back references to named subpatterns use the Python syntax (?P=name). We could rewrite the above example as follows:

  (?<p1>(?i)rah)\s+(?P=p1)

There may be more than one back reference to the same subpattern. If a subpattern has not actually been used in a particular match, any back references to it always fail. For example, the pattern

  (a|(bc))\2

always fails if it starts to match "a" rather than "bc". Because there may be many capturing parentheses in a pattern, all digits following the backslash are taken as part of a potential back reference number. If the pattern continues with a digit character, some delimiter must be used to terminate the back reference. If the PCRE_EXTENDED option is set, this can be whitespace. Otherwise an empty comment can be used.

A back reference that occurs inside the parentheses to which it refers fails when the subpattern is first used, so, for example, (a\1) never matches. However, such references can be useful inside repeated subpatterns. For example, the pattern

  (a|b\1)+

matches any number of "a"s and also "aba", "ababbaa" etc. At each iteration of the subpattern, the back reference matches the character string corresponding to the previous iteration. In order for this to work, the pattern must be such that the first iteration does not need to match the back reference. This can be done using alternation, as in the example above, or by a quantifier with a minimum of zero.

ASSERTIONS

An assertion is a test on the characters following or preceding the current matching point that does not actually consume any characters. The simple assertions coded as \b, \B, \A, \G, \Z, \z, ^ and $ are described above. More complicated assertions are coded as subpatterns. There are two kinds: those that look ahead of the current position in the subject string, and those that look behind it.

An assertion subpattern is matched in the normal way, except that it does not cause the current matching position to be changed. Lookahead assertions start with (?= for positive assertions and (?! for negative assertions. For example,

  \w+(?=;)

matches a word followed by a semicolon, but does not include the semicolon in the match, and

  foo(?!bar)

matches any occurrence of "foo" that is not followed by "bar". Note that the apparently similar pattern

  (?!foo)bar

does not find an occurrence of "bar" that is preceded by something other than "foo"; it finds any occurrence of "bar" whatsoever, because the assertion (?!foo) is always true when the next three characters are "bar". A lookbehind assertion is needed to achieve this effect.

If you want to force a matching failure at some point in a pattern, the most convenient way to do it is with (?!) because an empty string always matches, so an assertion that requires there not to be an empty string must always fail.

Lookbehind assertions start with (?<= for positive assertions and (?<! for negative assertions. For example,

  (?<!foo)bar

does find an occurrence of "bar" that is not preceded by "foo". The contents of a lookbehind assertion are restricted such that all the strings it matches must have a fixed length. However, if there are several alternatives, they do not all have to have the same fixed length. Thus

  (?<=bullock|donkey)

is permitted, but

  (?<!dogs?|cats?)

causes an error at compile time. Branches that match different length strings are permitted only at the top level of a lookbehind assertion. This is an extension compared with Perl (at least for 5.8), which requires all branches to match the same length of string. An assertion such as

  (?<=ab(c|de))

is not permitted, because its single top-level branch can match two different lengths, but it is acceptable if rewritten to use two top-level branches:

  (?<=abc|abde)

The implementation of lookbehind assertions is, for each alternative, to temporarily move the current position back by the fixed width and then try to match. If there are insufficient characters before the current position, the match is deemed to fail.

PCRE does not allow the \C escape (which matches a single byte in UTF-8 mode) to appear in lookbehind assertions, because it makes it impossible to calculate the length of the lookbehind.

Atomic groups can be used in conjunction with lookbehind assertions to specify efficient matching at the end of the subject string. Consider a simple pattern such as

  abcd$

when applied to a long string that does not match. Because matching proceeds from left to right, PCRE will look for each "a" in the subject and then see if what follows matches the rest of the pattern. If the pattern is specified as

  ^.*abcd$

the initial .* matches the entire string at first, but when this fails (because there is no following "a"), it backtracks to match all but the last character, then all but the last two characters, and so on. Once again the search for "a" covers the entire string, from right to left, so we are no better off. However, if the pattern is written as

  ^(?>.*)(?<=abcd)

or, equivalently,

  ^.*+(?<=abcd)

there can be no backtracking for the .* item; it can match only the entire string. The subsequent lookbehind assertion does a single test on the last four characters. If it fails, the match fails immediately. For long strings, this approach makes a significant difference to the processing time.

Several assertions (of any sort) may occur in succession. For example,

  (?<=\d{3})(?<!999)foo

matches "foo" preceded by three digits that are not "999". Notice that each of the assertions is applied independently at the same point in the subject string. First there is a check that the previous three characters are all digits, and then there is a check that the same three characters are not "999". This pattern does not match "foo" preceded by six characters, the first of which are digits and the last three of which are not "999". For example, it doesn't match "123abcfoo". A pattern to do that is

  (?<=\d{3}...)(?<!999)foo

This time the first assertion looks at the preceding six characters, checking that the first three are digits, and then the second assertion checks that the preceding three characters are not "999".

Assertions can be nested in any combination. For example,

  (?<=(?<!foo)bar)baz

matches an occurrence of "baz" that is preceded by "bar" which in turn is not preceded by "foo", while

  (?<=\d{3}(?!999)...)foo

is another pattern which matches "foo" preceded by three digits and any three characters that are not "999".

Assertion subpatterns are not capturing subpatterns, and may not be repeated, because it makes no sense to assert the same thing several times. If any kind of assertion contains capturing subpatterns within it, these are counted for the purposes of numbering the capturing subpatterns in the whole pattern. However, substring capturing is carried out only for positive assertions, because it does not make sense for negative assertions.

CONDITIONAL SUBPATTERNS

It is possible to cause the matching process to obey a subpattern conditionally or to choose between two alternative subpatterns, depending on the result of an assertion, or whether a previous capturing subpattern matched or not. The two possible forms of conditional subpattern are

  (?(condition)yes-pattern)

  (?(condition)yes-pattern|no-pattern)

If the condition is satisfied, the yes-pattern is used; otherwise the no-pattern (if present) is used. If there are more than two alternatives in the subpattern, a compile-time error occurs.

There are three kinds of condition. If the text between the parentheses consists of a sequence of digits, the condition is satisfied if the capturing subpattern of that number has previously matched. The number must be greater than zero. Consider the following pattern, which contains non-significant white space to make it more readable (assume the PCRE_EXTENDED option) and to divide it into three parts for ease of discussion:

  ( \( )?    [^()]+    (?(1) \) )

The first part matches an optional opening parenthesis, and if that character is present, sets it as the first captured substring. The second part matches one or more characters that are not parentheses. The third part is a conditional subpattern that tests whether the first set of parentheses matched or not. If they did, that is, if subject started with an opening parenthesis, the condition is true, and so the yes-pattern is executed and a closing parenthesis is required. Otherwise, since no-pattern is not present, the subpattern matches nothing. In other words, this pattern matches a sequence of non-parentheses, optionally enclosed in parentheses.

If the condition is the string (R), it is satisfied if a recursive call to the pattern or subpattern has been made. At "top level", the condition is false. This is a PCRE extension. Recursive patterns are described in the next section.

If the condition is not a sequence of digits or (R), it must be an assertion. This may be a positive or negative lookahead or lookbehind assertion. Consider this pattern, again containing non-significant white space, and with the two alternatives on the second line:

  (?(?=[^a-z]*[a-z])

  \d{2}-[a-z]{3}-\d{2}  |  \d{2}-\d{2}-\d{2} )

The condition is a positive lookahead assertion that matches an optional sequence of non-letters followed by a letter. In other words, it tests for the presence of at least one letter in the subject. If a letter is found, the subject is matched against the first alternative; otherwise it is matched against the second. This pattern matches strings in one of the two forms dd-aaa-dd or dd-dd-dd, where aaa are letters and dd are digits.

COMMENTS

The sequence (?# marks the start of a comment which continues up to the next closing parenthesis. Nested parentheses are not permitted. The characters that make up a comment play no part in the pattern matching at all.

If the PCRE_EXTENDED option is set, an unescaped # character outside a character class introduces a comment that continues up to the next newline character in the pattern.

Last updated: 03 February 2004

Copyright © 1997-2006 University of Cambridge.

UTF-8 SUPPORT

Starting at release 3.3, PCRE has had some support for character strings encoded in the UTF-8 format. For release 4.0 this has been greatly extended to cover most common requirements.

The following comments apply when PCRE is running in UTF-8 mode:

1. PCRE assumes that the strings it is given contain valid UTF-8 codes. It does not diagnose invalid UTF-8 strings. If you pass invalid UTF-8 strings to PCRE, the results are undefined.

2. In a pattern, the escape sequence \x{...}, where the contents of the braces is a string of hexadecimal digits, is interpreted as a UTF-8 character whose code number is the given hexadecimal number, for example: \x{1234}. If a non-hexadecimal digit appears between the braces, the item is not recognized. This escape sequence can be used either as a literal, or within a character class.

3. The original hexadecimal escape sequence, \xhh, matches a two-byte UTF-8 character if the value is greater than 127.

4. Repeat quantifiers apply to complete UTF-8 characters, not to individual bytes, for example: \x{100}{3}.

5. The dot metacharacter matches one UTF-8 character instead of a single byte.

6. The escape sequence \C can be used to match a single byte in UTF-8 mode, but its use can lead to some strange effects.

7. The character escapes \b, \B, \d, \D, \s, \S, \w, and \W correctly test characters of any code value, but the characters that PCRE recognizes as digits, spaces, or word characters remain the same set as before, all with values less than 256.

8. Case-insensitive matching applies only to characters whose values are less than 256. PCRE does not support the notion of "case" for higher-valued characters.

9. PCRE does not support the use of Unicode tables and properties or the Perl escapes \p, \P, and \X.

DIFFERENCES FROM PERL

This document describes the differences in the ways that PCRE and Perl handle

regular expressions. The differences described here are with respect to Perl

5.8.

1. PCRE does not allow repeat quantifiers on lookahead assertions. Perl

permits them, but they do not mean what you might think. For example, (?!a){3}

does not assert that the next three characters are not "a". It just asserts that

the next character is not "a" three times.

2. Capturing subpatterns that occur inside negative lookahead assertions are

counted, but their entries in the offsets vector are never set. Perl sets its

numerical variables from any such patterns that are matched before the assertion

fails to match something (thereby succeeding), but only if the negative

lookahead assertion contains just one branch.

3. Though binary zero characters are supported in the subject string, they

are not allowed in a pattern string because it is passed as a normal C string,

terminated by zero. The escape sequence "

\0"

can be used in the pattern to

represent a binary zero.

4. The following Perl escape sequences are not supported: \l, \u, \L, \U, \P,

\p, \N, and \X. In fact these are implemented by Perl's general string-handling

and are not part of its pattern matching engine. If any of these are encountered

by PCRE, an error is generated.

5. PCRE does support the \Q...\E escape for quoting substrings. Characters in

between are treated as literals. This is slightly different from Perl in that $

and @ are also handled as literals inside the quotes. In Perl, they cause

variable interpolation (but of course PCRE does not have variables). Note the

following examples:

    Pattern            PCRE matches      Perl matches

    \Qabc$xyz\E        abc$xyz           abc followed by the contents of $xyz

    \Qabc\$xyz\E       abc\$xyz          abc\$xyz

    \Qabc\E\$\Qxyz\E   abc$xyz           abc$xyz

In PCRE, the \Q...\E mechanism is not recognized inside a character class.

8. Fairly obviously, PCRE does not support the (?{code}) and (?p{code}) constructions. However, there is some experimental support for recursive patterns using the non-Perl items (?R), (?number) and (?P>name). Also, the PCRE "callout" feature allows an external function to be called during pattern matching.

9. There are some differences that are concerned with the settings of captured strings when part of a pattern is repeated. For example, matching "aba" against the pattern /^(a(b)?)+$/ in Perl leaves $2 unset, but in PCRE it is set to "b".

10. PCRE provides some extensions to the Perl regular expression facilities:

(a) Although lookbehind assertions must match fixed length strings, each alternative branch of a lookbehind assertion can match a different length of string. Perl requires them all to have the same length.

(b) If PCRE_DOLLAR_ENDONLY is set and PCRE_MULTILINE is not set, the $ meta-character matches only at the very end of the string.

© If PCRE_EXTRA is set, a backslash followed by a letter with no special meaning is faulted.

(d) If PCRE_UNGREEDY is set, the greediness of the repetition quantifiers is inverted, that is, by default they are not greedy, but if followed by a question mark they are.

(e) PCRE_ANCHORED can be used to force a pattern to be tried only at the first matching position in the subject string.

(f) The PCRE_NOTBOL, PCRE_NOTEOL, PCRE_NOTEMPTY, and PCRE_NO_AUTO_CAPTURE options for pcre_exec() have no Perl equivalents.

(g) The (?R), (?number), and (?P>name) constructs allows for recursive pattern matching (Perl can do this using the (?p{code}) construct, which PCRE cannot support.)

(h) PCRE supports named capturing substrings, using the Python syntax.

(i) PCRE supports the possessive quantifier "++" syntax, taken from Sun's Java package.

(j) The (R) condition, for testing recursion, is a PCRE extension.

(k) The callout facility is PCRE-specific.

Last updated: 03 February 2004

Copyright © 1997-2006 University of Cambridge.


EasyPatterns

 EasyPatterns
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

EasyPatterns™ are a revolutionary new way to describe text patterns that is easy to understand, easy to use, and still just as powerful as the old-hard to understand Perl-style or grep-style pattern matching languages.

What is a Pattern?

A pattern describes text in a general way. Searching based on literal text yields only one match, whereas searching with a pattern can match a whole class of text that shares the specified characteristics. Phone numbers (in North America) are a perfect example: three groups of digits separated by various punctuation characters. Here's a simple Text Machine pattern that will find most phone numbers:

[3 digits, punctuation, 3 digits, punctuation, 4 digits]

Let's suppose that you have a list of addresses phone numbers gathered from different sources. The phone numbers are likely to be in many formats: some with / and -, some with periods, etc. To convert them to a consistent format you need to keep the three groups of digits and insert new punctuation in between. With TextPipe's EasyPatterns, this is as easy as

Search for: "[punctuation capture(3 digits) punctuation capture(3 digits) punctuation capture(4 digits)]"

Replace with: "$1-$2-$3"

This section introduces all of the essential elements of EasyPattern patterns. If you are new to pattern-matching, there's no reason you have to read the entire section at once. Feel free to stop after any section and continue with the next 2 sections or spend time reviewing example. Also, don't worry if you don't immediately understand every point. The only way to learn pattern matching is to create your own patterns, including the inevitable trial and error required to get just the right match. TextPipe's trial run area was designed with this in mind.

Throughout this text and the reference section, double quotes are used around both the pattern and the example text that it would match. The double quotes should *NOT* be entered in TextPipe.

Basics

Unlike other pattern matching languages, EasyPatterns use plain English keywords such as "letter" and "digit" to define text patterns. At the most basic level, each keyword appears in [square brackets], e.g.

"[letter][digit]" « matches "a0" ... "a9", "b0" ... "b9", etc.

Why use brackets? Patterns can also include literal text; brackets distinguish that from the pattern specification:

"abc[digit]" « matches "abc0", "abc1" ... "abc9"

"Z[letter][letter]" « matches "Zoo", "Zed", etc.

"whatever" « keywords are not required, EasyPattern can search just for literal text

In addition to simple keywords, the brackets can also contain expressions. The most common expressions indicate repetition, e.g.

"[optional digit]" « 0 or 1

"[oneOrMore digits]" « 1, 2 or many (no spaces are allowed in "oneOrMore")

"[1+ digits]" « same meaning, different notation

"[3+ digits]" « 3 or more

"[3 digits]" « exactly 3

"[3 to 5 digits]" « 3, 4 or 5

Simple keywords and expressions can be combined, e.g. here's a pattern that matches many North American telephone numbers:

"[3 digits][punctuation][3 digits][punctuation][4 digits]"

That's a lot of brackets! The above pattern works, but EasyPattern also allows multiple keywords and expressions to be combined with commas, e.g.

"[3 digits, punctuation, 3 digits, punctuation, 4 digits]"

In fact, even the commas are optional, though you may find they make the pattern easier to read.

"[3 digits punctuation 3 digits punctuation 4 digits]"

Many patterns require mixing keywords with literal text, e.g. to find telephone numbers in a certain local exchange:

"978[punctuation]692[punctuation, 4 digits]"

That pattern works fine; EasyPattern also lets you include the literal text inside the bracketed expression using single quotes:

"['978', punctuation, '692', punctuation, 4 digits]"

Character Sets

One of the fundamental ideas in pattern matching is that each character belongs to one or more character sets. For example, "a" is a member of the set of letters, represented by the keyword "[letter]". "2" is a member of the set "[digit]", etc. In EasyPattern, character sets can be combined with "or". For example, both "a" and "2" are members of the combined set "[letter or digit]". You can combine any number of sets with "or", e.g. "[letter or punctuation or symbol or whitespace]" is nearly everything except digits.

You can also use "or" to combine individual characters into a set, e.g. "[space or tab]". See the EasyPattern Reference for many additional whitespace options.

Tip: Parentheses are not required. For example, "[1+ space or tab]" and "[1+ (space or tab)]" have the same meaning.

The keyword "[char]" or "[character]" represents the set of all characters, including letters, digits, punctuation, symbols, space, tab, return, linefeed, formfeed and control characters. (NULL is not included; see EasyPattern Reference for details and a workaround.) Sometimes you want to match every character in a paragraph, i.e. every character except return (or formfeed). EasyPattern has a special keyword "[paragraphChar]" for this situation. For example, given the text "abc" then a return then "123":

"[1+ char]" « matches all 7 characters

"[1+ paragraphChar]" « matches the first 3 characters

Tip: Look up "[paragraphChar]" in the EasyPattern Reference; you'll find a whole family of keywords for words, columns, lines & paragraphs.

Sometimes it's easier to specify the set that you don't want rather than listing everything you want:

"[not letter]" « match anything except a letter "[not letter or digit]" « match anything except a letter or digit "[1+ not letter or digit]" « matches a sequence of characters Tip: Since "not" is processed after the character set use of "or", parentheses are still optional. For example, "[not space or tab]" and "[not (space or tab)]" have the same meaning.

Alternatives

One of the most powerful features of EasyPattern isn't related to patterns per se: the ability to find a match from among two or more choices. Some examples:

"this[or]that"

"['this' or 'that']"

"EasyPattern[or]Player"

"StarTrek[or]StarWars[or]OSA[or]scripting"

Tip: If you include an alternative in a larger pattern, you must include parentheses:

"[digit ('this' or 'that')]"

The keyword "or" has two uses: combining character sets and listing alternatives. The differences are sometimes subtle though will become more clear with experience. See the EasyPattern Reference for details.

Groups

EasyPattern lets you group any part of a pattern with parentheses. Groups can also be preceded by capture() to capture the text that the group matched for later use, either in a replacement string (with $1, $2..$9, $a..$z), or in a later part of the pattern, e.g. [capture( ... ), group1].

There are many reasons to group parts of a pattern:

· Apply quantity or repetition to a group, e.g. [1+ (letter, digit)] will match "r2d2"
· Specify one of many alternatives with "or"
· Mark logical groups to make the pattern easier to read
· Extract a certain portion of a pattern (where the full pattern is required to provide context)
· Replace certain portions of a pattern, including changing the order

For example, you could change phone numbers from many different formats to your preferred format, searching for:

"[punctuation capture(3 digits) punctuation capture(3 digits) punctuation capture(4 digits)]"

and replacing with:

"$1-$2-$3"

Tip: Parentheses groups can be nested to any level, e.g. [1+ ((letter, digit) or (digit, letter))] will match "r2d2", "r22d", "2rd2", "2r2d", etc.

Case in Keywords

EasyPattern keywords are not allowed to contain spaces, and are not case sensitive; e.g. [uppercaseLetter], [UppercaseLetter], [UPPERCASELETTER], [UppErcAsELEttEr] are equivalent.

Case in Literals

Although keywords are not case sensitive, the case sensitivity of literal text is controlled by TextPipe 'Match Case' option.

Patterns in Replace Strings

In the simplest replacement, you will find a pattern and replace it with literal text, e.g.

replace "*[or]_" with "-"

To add literal text before or after existing text, use $0 to represent the entire found text, e.g.

replace "-[space]" with "\t$0" -- insert a tab before list items

The phone number example shows a common use of [(...)#] and $# (where # is a number from 0-9); here's another example:

replace "[capture(column) tab capture(column)]" with "$2\t$1" -- swap columns

What Next?

EasyPattern's core vocabulary was covered in this section. The best way to learn is by creating patterns for real-world tasks. When you have questions on the basics, refer back to this section. If you need a more detailed explanation of any topic or are looking for additional keywords to handle a new case, see the EasyPattern Reference.


EasyPatterns Reference

 EasyPatterns Reference
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Note: For the latest copy of this documentation, please see the complete EasyPatterns reference (online)

Contents - EasyPatterns 2.8

· Specifying literal text/static text
· Common character classes such as letters and digits
· Combining character classes and creating your own character classes
· Handling alternatives - A or B
· How many repeats? Quantity, repetition and optional pieces
· Greediness, or shortest match vs longest match, atomic matching
· Grouping text and capturing text for use in a replace string
· Commenting your patterns for readability
· Patterns for whitespace
· More Keywords
· Advanced - operator precedence, order of operations

Conventions: Actual EasyPatterns are highlighted with [...].

Specifying literal text/static text

The only character that is 'special' is the left square bracket or [. The simplest pattern is just literal text, with no left square brackets. Whenever we need EasyPattern keywords we just put them inside [...].

EasyPattern | Description | Matches this text...
hello there | No [ ... ] expression has been used, this is just literal text | hello there
hello there [longest 1 or more letters] | The special part is [longest 1 or more letters] | hello there Fred, hello there Cornelia, etc
I am [1 or more digits] years old] | The special part is [1 or more digits] | I am 2 years old, I am 302 years old, etc
This is a left square bracket [ '[' ] | This shows how to insert a left square bracket in literal text. | This is a left square bracket [

To use multiple keywords, you can either

Put them next to each other | [...][...] | e.g. [letter][digit] matches "a1", "b1", etc.
Put commas between them | [..., ...] | e.g. [letter, digit] instead of [letter][digit]
Put spaces between them | [... ...] | e.g. [letter digit] instead of [letter][digit]

You can also put literal text anywhere inside [...] using single quotes or double quotes.

['literal'] | e.g. "['abc']" instead of "abc"
[... 'literal'] | e.g. "[digit, 'abc']" instead of "[digit]abc"
['literal' ...] | e.g. "['abc', digit]" instead of "abc[digit]"
[... 'literal' ...] | e.g. "[digit, 'abc', digit]" instead of "[digit]abc[digit]"
· There is usually no difference in meaning between including literal text within a bracketed expression (in single quotes) and leaving literal text outside the brackets. The choice is a matter of individual preference. One exception: when using [not], only the single-quoted literal will work, e.g. [not '-'].

Common character classes such as letters and digits

The most important keywords represent character classes or sets, that is, a set of related characters.

Any character, letters, digits, etc.
[character], [char], [chars], [characters] | All 256 chars (every character including NULL). EasyPattern's [character] or [char] will match any character including return. If you want any character except a return (or formfeed), use [paragraphChar]; that is, any character that could appear in a paragraph. Details below.
[letter], [letters] | Includes é and ü, common in certain European languages.
[digit], [digits] | Decimal digits 0-9
[number], [numbers], [numeric] | A number with an optional leading sign, digits, optional decimal point and trailing digits
[Integer] | A number with an optional leading sign, followed by digits
[Float] | A number with an optional leading sign, digits, optional decimal point and trailing digits, optionally followed by 'e', a sign, and 1 or more digits
[hexdigit] | Hexadecimal digits 0-9, a-f and A-F
[EBCDICletter] | An EBCDIC letter
[EBCDICupper] | An EBCDIC uppercase letter
[EBCDIClower] | An EBCDIC lowercase letter
[EBCDICdigit] | An EBCDIC digit, ASCII F0-F9
[punctuation] | Printing characters, excluding letters and digits, includes !?.,:; " ' ' / - () {} - Note that « and » are considered punctuation.
[symbol], [symbols] | ~@#$%^&* EasyPattern distinguishes punctuation from symbols; the sets do not overlap. For broader combinations, see [printableChar] and [typewriterChar]. For narrower focus, see [sentencePunctuation], [anyQuote], [anyBracket] and [anyDash].
Special letters
[upper], [uppercase], [uppercaseLetter] | Uppercase letters. Note: In TextPipe you will also need to enable the Match Case option for this to make any difference.
[lower], [lowercase], [lowercaseLetter] | Lowercase letters. Note: In TextPipe you will also need to enable the Match Case option for this to make any difference.
Reserved punctuation
[leftBracket] | [
[rightBracket] | ]
[leftParen], [leftParenthesis] | (
[rightParen], [rightParenthesis] | )
[leftAngle], [lessThan] | <
[rightAngle], [greaterThan] | >
[comma] | ,
[singleQuote] | '
[doubleQuote], [quote] | " (i.e. standard ASCII "straight" quotation mark)
[graveAccent] | ` (previously incorrectly called backwardSingleQuote)
ASCII function()
asc(code, ...), ascii(code, ...) | The ASCII() or ASC() function embeds arbitrary control characters by entering the control code in decimal or hex (precede the hex digit with '$" eg $ff). You can add one or more control characters by separating each with a space or comma. e.g. ASC( 65, 66 ) outputs 'AB' into the pattern.
EBCDIC function()
ebcdic( literal ) | The EBCDIC() function embeds Mainframe EBCDIC characters translated from a string literal you provide e.g. EBCDIC( '0' ) outputs \xF0 (an EBCDIC '0')
· Square brackets are "reserved" by EasyPattern. The other punctuation marks listed here are literal when they appear outside of brackets.
· EasyPattern gives special meaning to certain punctuation marks; these keywords can be used to represent the literal character.
· A literal comma and left & right brackets & parentheses may appear inside single quotes. The keywords are provided to make patterns easier to read.

Filename patterns

[Drive] | A drive letter followed by a colon (:) e.g d:\special\folder\filename.doc, feeding the letter into @Drive@
[Folder] | A path fragment between \ ... \, e.g. d:\special\folder\filename.doc, feeding into @Folder@
[Path] | A path with optional drive e.g. e.g. d:\special\folder\filename.doc, feeding into @Drive@ and @Path@
[UNCpath] | A UNC path consisting of server, share and path - these feed into @Server@, @Share@ and @Path@
[Filename] | A filename, starting from \ and not ending with \ e.g d:\special\folder\filename.doc

Combining character classes and creating your own character classes

There are many ways to create your own character sets to match exactly the characters you require.

You can combine existing character sets using "or":

[... or ...]

e.g. [letter or digit], ['a' or 'b']
· Most EasyPattern keywords refer to a set of characters from which one will match. The first use of "or" is to make a larger set -- though again, only one of the larger set will match. (Any quantity can be specified using repetition keywords, but it is still applying a quantity to a single character not to multiple characters)
· When sets are combined with "or", parentheses are optional. (In technical terms, the character set use of "or" has very high precedence; see below)

[letter, letter or digit, letter] -- matches "aaa", "xyz", "h4q", "b7f" etc.

It doesn't hurt to add parentheses even though they are not required.

[letter (letter or digit) letter] -- same as above

Negation - Match anything except a given set

Instead of specifying all the characters that could occur in a match, it is often convenient to specify characters that could not occur.

[not ...], [non ...], [anyExcept ...]

e.g. [oneOrMore non letter]
· EasyPattern has keywords for [quotedString] and [HTMLTag], but if it didn't, they would be easy to define:

['<', oneOrMore not '>', '>'] « same as [HTMLTag]

[quote, oneOrMore not quote, quote] « a simple definition for [quotedString]

· Negation can only be applied to a single character, or a character set from which one will match. For example:

[not letter] « fine

[not letter or digit] « fine: [letter or digit] is a set from which one will match

[not word] « ERROR: "word" matches multiple characters

[not 'a'] « fine (a single character)

[not 'whatever'] « ERROR

[not lineChar or letter] « CAREFUL! [lineChar] is defined as [not linefeed OR verticalTab OR formfeed OR return]. You cannot combine negated and non-negated characters sets, so this pattern is equivalent to [ (not lineChar) or letter ], instead of [ not (lineChar or letter) ]

Custom Sets

Keywords such as [letter] and [digit] are character sets defined internally to EasyPattern; the angle bracket notation lets you define your own characters sets. In both case, EasyPattern matches any single character in that set.

[<...>]

e.g. [<aeiou>], [<135>], [<!@#$%^&*>]
· For single characters, [or] and a set are interchangeable, e.g. [<aei>] and ['a' or 'e' or 'i'] have the same meaning.
· User defined sets, single character literals and EasyPattern keywords can be combined with [or]:

[<aeiou> or <123> or '7' or symbol]

Handling Alternatives - A or B

alternative patterns with "or"

[... or ...]

e.g. ['Player' or 'EasyPattern']
· When [or] is used to specify alternatives as part of a larger pattern, grouping parentheses are required, e.g.

[space, 'Player' or 'EasyPattern', space] -- may not mean what you think!

[space]Player[or]EasyPattern[space] -- may not mean what you think!

[(space, 'Player') or ('EasyPattern', space)] -- that's what they mean

[space ('Player' or 'EasyPattern') space] -- this might be what you wanted

Remember: as noted in the section on expressions, commas are allowed between items to make patterns easier to read; they do not affect what the pattern means.

· If you leave out the parentheses, EasyPattern will treat everything to the left of the [or] as one implicit group and everything to the right of the [or] as a separate group. Note that visual grouping with brackets or commas is not enough; you must use parentheses. For example, all of the following will be interpreted as [(digit, 'this') or ('that')]:

[digit, 'this' or 'that'] « careful; the commas may mislead

[digit]['this' or 'that'] « careful; the brackets may mislead

[digit 'this' or 'that'] « the grouping isn't clear; parentheses would help

[digit]this[or]that « the grouping isn't clear; parentheses would help

As noted in the previous section, parentheses are not required when [or] is used to combine character sets.

"or" as set vs. "or" as alternative

In many cases, you don't have to worry that there are two different uses for "or"; both generally make sense in context. However, there are 2 reasons for learning the differences:

· or as set doesn't require parentheses; the grouping is implied
· or as set can be part of a "not" expression since it still represents one character

How many repeats? Quantity, repetition and optional pieces

Notation: "..." is any appropriate keyword or expression, # is a number (one or more digits; the maximum varies with context).

repetition | examples | will match...
[optional ...], [zeroOrOne ...] | [digit, optional letter], [digit, zeroOrOne letter] | 2, 2a
[0+ ...], [zeroOrMore ...] | [digit, zeroOrMore letters] | 2, 2a, 2aa, 2aaa, 2aaaa...
[1+ ...], [oneOrMore ...] | [digit, oneOrMore letters] | 2a, 2aa, 2aaa, 2aaaa...
[2+ ...], [many ...], [twoOrMore ...] | [digit, many letters], [digit, twoOrMore letters] | 2aa, 2aaa, 2aaaa...
[#+ ...] | [digit, 5+ letters] | 2aaaaa, 2aaaaaa...
· A space is not allowed, e.g. [one or more] will not be recognized
· The words are all special cases, e.g. [threeOrMore ...] will not work (use [3+ ...])
specific quantity, quantity range (where # is a number) |  | will match...
[# ...] | [5 letters] | aaaaa, bbbbb
[# to # ...] [# - # ...] [# .. # ...] | [3 to 5 letters] [3-5 letters] [3..5 letters] | aaa, aaaa, aaaaa

Quantities can now be entered in Hex form by preceding them with '$' e.g $ff.

Greediness, or shortest match vs longest match, atomic matching

When the repetition or count includes a range of values to match, EasyPattern has the choice of matching the "shortest" sequence of characters that fits the pattern, or the "longest" that fits the pattern. For example

[shortest zeroOrOne ...] | 0 or 1 | will try to match zero occurrences
[shortest zeroOrMore ...] | 0+ | will try to match zero occurrences
[shortest oneOrMore ...] | 1+ | will try to match one occurrence
[shortest twoOrMore ...] | 2+ | will try to match two occurrences

EasyPattern defaults to the SHORTEST match so the "shortest" keyword is optional.

[shortest ... ...] | match the lowest possible number of repetitions (default)
[longest ... ...] | match the highest possible number of repetitions
· In these cases, EasyPattern will only match more than the minimum if required to complete additional parts of the pattern, e.g. given "abc123" and the pattern [shortest oneOrMore letter, digit], EasyPattern will match "abc1", i.e. all 3 letters. However, given the same string and the pattern [shortest oneOrMore letter], EasyPattern will just match "a" the first letter. Given the same string and [longest oneOrMore letter], EasyPattern will match "abc". Note that EasyPattern always starts with the first character that matches that pattern, e.g. despite "c1" being shorter than "abc1", EasyPattern matches the latter.
· Shortest/longest can be confusing.
· Shortest can be quite slow, use "not" if possible

Pattern matching can become very time consuming if the number of repeats is not known. Take for example

Pattern: a[ 1+ digits ]b Matching text: a22222222z

The pattern matcher first matches a, and 8 '2's, then it finds that 'z' does not match 'b'. So it backtracks, trying with 7 '2's, failing again, then with 6 '2's, all the way back to 1 '2', before finally giving up, and starting to test for 'a' again.

If we know that backtracking into a repeated match will still result in failure, we can tell EasyPatterns to not bother, by using the atomic keyword.

Pattern: a[ atomic(1+ digits) ]b Matching text: a22222222z

This time, the pattern matcher first matches a, and 8 '2's, then it finds that 'z' does not match 'b'. So it backtracks all the way back to starting to test for 'a' again.

Literals, groups

All of the repetition & quantity keywords can be applied to literals and groups as well as to individual keywords, e.g.

[oneOrMore 'ab'] « matches "ab", "abab", "ababab" etc.

[oneOrMore letter or digit] « matches "aaa", "456", "a45bbb" etc.

[oneOrMore not letter or digit] « matches punctuation, symbols, whitespace etc.

[oneOrMore ('alpha' or 'omega')] « matches "alphaalapha", "alphaomega" etc.

[oneOrMore (letter, digit)] « matches "r2", "r2d2", "r2d2f7b2c4" etc.

Grouping text and capturing text for use in a replace string

[(...)] | A non-capturing group
[capture(...)] | Assigns the contents of the group to a variable which can be referred to later in both the search pattern ([group#] e.g. [group6] ,# can range from 1 to 26) and in the replacement string ($# e.g. $6, # can range from 1-9, a-z. $0 represents the entire matched string).
[group#] | Matches the same text that a previously captured group found. [capture(letter), group1] « matches "ee", "bb", "cc" etc
[mustBeginWith(...) ...], [mustNotBeginWith(...) ...] | When a match is found, it must be/must not be preceded by what is in the brackets. The bracket contents are NOT included in the actual match. The bracket contents are limited to fixed length strings - so no '3+' etc are allowed. This must be the first part of your pattern. [mustBeginWith( 'hello' or 'goodbye' ) 'fred']
[... mustEndWith(...)], [... mustNotEndWith(...)] | When a match is found, it must be/must not be followed by what is in the brackets. The bracket contents are NOT included in the actual match. The bracket contents are limited to fixed length strings - so no '3+' etc are allowed. This must be the last part of your pattern. ['fred' mustEndWith( 'erick' or 'dy' ) ]
· Parentheses must match, i.e. ")" always ends the most recent "(", independent of number.

Commenting your patterns for readability

EasyPattern allows comments to be included in multi-line patterns using the

character ';' or '#' to make the start of a comment, extending until the end of

the line e.g.

[ 3 space ;look for 3 spaces

'hello' #then the keyword we want

]

Patterns for Whitespace

[space], [spaces] | ASCII 32
[nonbreakingSpace] | ASCII 202
[whitespace] | [space OR tab OR cr OR lf OR verticalTab OR nonbreakingSpace]
[tab] | ASCII 9, \t
[return], [cr] | ASCII 13, \r
[linefeed], [lf] | ASCII 10, \n
[verticalTab] | ASCII 11
[formfeed] | ASCII 12, \f
[null] | ASCII 0
[CRLF] | [return, linefeed]
[newline] | [(return, linefeed) or return or linefeed]
[DOSNewline] | [return, linefeed]
[UNIXNewline] | [linefeed]
[MacNewline] | [return]
· [not] cannot be applied to [CRLF], [newline] or [DOSNewline] since they either are or may be a character sequence rather than just a single character.
· A space character can usually be typed directly into a pattern ([ ' ' ]) but using the keyword may make the pattern easier to understand (and modify later)

Whitespace combinations

[horizontalWhitespace], [hSpace] | [space or nonbreakingSpace or tab]
[verticalWhitespace], [vSpace] | [return or linefeed or formfeed or vertical tab]
words, columns, lines & paragraphs
[wordDelimiter] | [space OR tab OR linefeed OR verticalTab OR formfeed OR return]
[wordChar] | [not wordDelimiter]
[word] | [1+ wordChar]
[columnDelimiter] | [tab OR linefeed OR formfeed OR return]
[columnChar] | [not columnDelimiter]
[column] | [1+ columnChar] Note: Use [0+ columnChar] instead if the column could be blank
[lineDelimiter] | [linefeed OR verticalTab OR formfeed OR return]
[lineChar] | [not lineDelimiter]
[line] | [1+ lineChar] Note: Use [0+ lineChar] instead if the line could be blank
[paragraphDelimiter] | [formfeed OR return]
[paragraphChar] | [not paragraphDelimiter]
[paragraph] | [1+ paragraphChar]
· The above delimiters are characters not positions; they will "consume" the character that they match. In contrast, [TextStart] and [TextEnd] (below) are positions.
· The above objects (word, column, line, paragraph) do not include delimiters. So, to match multiple objects, you need to include the delimiters, e.g.

[2+ word] -- won't match anything

[2+ (word, optional wordDelimiter)] -- correct

· The definition for word is based strictly on whitespace so it will include punctuation, matching text such as "$27.52" and "fancy+name". Although in many cases it would be nice to exclude trailing punctuation, that pattern would fail for text like "S.M.U.". When EP's definition of a word isn't appropriate for your text, simply use the custom pattern that fits. For example, [1+ wordChar, letter or digit or symbol] would ensure that the last char is not punctuation.
· Word, column, line & paragraph require one or more character. If a line might be empty, use: [0+ lineChar] instead of [line].
· Because the definitions for word, column, line & paragraph look for anything except the appropriate delimiter (rather than the leading delimiter, a series of anything else, and the trailing delimiter), they can be used to get the rest of a word, column, line & paragraph when the starting point is already in the middle.
· These definitions allow control characters (except the specific whitespace used as delimiters) to appear in words, columns, lines & paragraphs.
· A column may contain the verticalTab character (it's used by FileMaker to indicate line breaks within a field)
· Word, column, line & paragraph consist of multiple characters so patterns like [not word] don't make sense.

Positions

[textStart] | matches at start of entire text
[textEnd] | matches at end of the entire text or before newline at end
[lineStart] | matches the start of a line (*)
[lineEnd] | matches the end of a line (*)
[wordBoundary], [wordBreak] | matches at a word boundary
[notWordBoundary] | matches when not at a word boundary

(*) [lineStart] and [lineEnd] will work fine if the file you're editing has Unix end of line characters, because the core EasyPattern engine assumes this. For DOS or Windows files, you should use

[ cr lf or textEnd ]

or

[ mustEndWith(cr lf or textEnd) ]

More Keywords

Combinations
[controlChar] | characters 0-31, 127 (careful: includes most whitespace)
[gremlin] | characters 0-31. The definition for [gremlin] is more cautious than in some products.
[printableChar] | [letter or digit or punctuation or symbol] (anything that prints ink on paper)
[typewriterChar] | [printableChar or space or tab or return] (excludes linefeed, vertical tab & formfeed)
Punctuation subsets (these items are included in [punctuation])
[sentencePunctuation] | .,;:!?¿¡
[anyBracket], [anyBrackets] | left/right paren/bracket/brace (i.e. "bracket" in the broad sense of the term)
[anyQuote] | [doubleQuote OR singleQuote OR backwardSingleQuote]
[dash], [hyphen] | - used interchangeably. we have adopted the common notion that these terms refer to the same character
[period] | .
[caret] | ^
[pound], [hash] | #
[slash] | /
[backslash] | \
[colon] | :
[percent] | %
[star], [asterisk] | *
[ampersand] | &
[pipe] | |
Real-world patterns
[HTMLTag] | <[1+ not '>']>
[HTMLStartTag] | <[not '/', 0+ not '>']> (i.e. any tag except an end tag)
[HTMLEndTag] | </[1+ not '>']>
[QuotedString] | [quote, 1+ ((backslash, quote) or not quote), quote]
[SocialSecurityNumber] | [3 digits, dash, 2 digits, dash, 4 digits]
[PhoneNumber] | Matches a US-style (xxx) xxx-xxxx number with a variety of punctuation marks. The matching text is captured into 3 successive $variables
[EmailAddress] | Matches email addresses. The name and domain parts are captured into 2 successive $variables
[IPAddress] | Matches numeric IP addresses. The matching text is captured into 4 successive $variables
[CreditCard] | Matches credit card numbers with a variety of punctuation marks. The matching text is captured into 4 successive $variables
[Hyperlink] | Matches a ftp, http, https, telnet, gopher or nntp internet url. The matching text is captured into 3 successive $variables
[DuplicateWord] | Matches a repeated word. The matching text is captured into 2 successive $variables
[PageNumber] | Matches a page number of the following forms: Page dd Page No dd Page No. dd Page Num. dd Pg Num dd Page Number dd. The matching text is captured into 3 successive $variables (Page, Number, #)
Data processing patterns (in TextPipe 6.8.2 and later)
[CSVfield] | A Comma-Separated-Value field. If fields are delimited by single or double quotes, embedded newlines are allowed, as are doubled-up quotes. The quotes are returned as part of the match.
[TABfield] | A Tab-delimited field. To process multiple tab fields e.g. [ 3 or more ( TABfield Tab) TABfield ]
[PipeField] | A Pipe-delimited field. To process multiple pipe fields e.g. [ 3 or more ( PipeField '|' ) PipeField ]
Date and time patterns
[Date] | Matches a date format DD-MM-YY or DD-MMM-YY
[AMPM] | The AM/PM part of a time
[Month] | A MonthName (English) or a MonthNumber
[MonthNumber] | 1-12, with an optional leading zero
[MonthName], [MonthNameShort], [MonthNameLong] | January-December and Jan-Dec (English)
[MonthNameLocal] | Full month names and 3 letter abbreviations for the current locale
[Day] | 1-31, with an optional leading zero
[DayNumber] | 01-31 (the leading zero is required)
[DayName], [DayNameShort], [DayNameLong] | Sunday-Saturday and Sun-Sat
[DayNameLocal] | Weekday names and 3 letter abbreviations for the current locale
[DayOfYear] | 1..366
[Year], [YearShort], [YearLong] | A 2 or 4 digit year (between 1800 and 2199)
[Hour] | A 12 or 24-hour hour, with optional leading zero
[Minute] | A 2 digit minute
[Second] | A 2 digit second

Using the real world patterns above, you can easily construct the following EasyPatterns:

HMS | [ Hour <:.-> Minute <:.-> Second ]
DMY | [ Day <-/ > Month <-/ > Year ]
MDY | [ Month <-/ > Day <-/ > Year ]
YMD | [ Year <-/ > Month <-/ > Day ]
Julian | [ Year DayOfYear ]
MY | [ Month <-/ > Year ]
MD | [ Month <-/ > Day ]
DM | [ Day <-/ > Month ]
HM | [ Hour <:. > Minute ]

Advanced - Operator precedence, Order of operations

A complete pattern may include many individual keywords and

many expressions. How do you know which keywords go together and where one expression stops and another begins? If in doubt, just enclose every expression in parentheses. But, EasyPattern has rules for combining keywords into expressions, so parentheses aren't always required. The traditional way of expressing these rules is to list the "precedence" of various operators or terms.:

· (...), including numbered groups
· [or] for characters sets and single-character literal
· [not]
· quantity specifiers ( oneormore, 2+, 3..7, etc)
· character set keywords (e.g. letter, digit) and single-character literals
· multi-character literals
· [or] as alternative, for groups and multi-character literals

Items with high precedence don't need parentheses; they group together automatically. For example, let's build a pattern step-by-step using the "high precedence" operators:

[letter or digit] « "or" for characters set keywords

[letter or digit or '.'] « and single-character literal

[letter or digit or '.' or <!?>] « and arbitrary set

[not letter or digit or '.' or <!?>] « reverse the meaning with not

[1+ not letter or digit or '.' or <!?>] « add a quantity specifier

[1+ (not (letter or digit or '.' or <!?>))] « if you like parentheses, though the meaning is the same

Adding lower precedence terms before, after or both doesn't change the grouping, though the expression is long enough that you may find a pair of commas, brackets, or parentheses helpful. As long as you understand how EasyPattern is doing the grouping, it doesn't matter whether you choose commas, brackets or parentheses. If the parentheses are added around something that is already a group, they don't change the meaning.

[punctuation 1+ not letter or digit or '.' or <!?> symbol]

[punctuation, 1+ not letter or digit or '.' or <!?>, symbol] « same meaning but easier to read

[punctuation][1+ not letter or digit or '.' or <!?>][symbol] « same meaning

[punctuation (1+ not letter or digit or '.' or <!?>) symbol] « same meaning

Remember, commas and brackets don't change the meaning, only the look. If you put them in the middle of high precedence terms, you might confuse yourself:

[punctuation 1+ not letter][or][digit or '.' or <!?> symbol] « same meaning but HARDER to read

[punctuation 1+ not letter, or, digit or '.' or <!?> symbol] « same meaning but HARDER to read

Only parentheses change the meaning:

[(punctuation 1+ not letter) or (digit or '.' or <!?> symbol)] « different meaning

Note that [or] for character sets and [or] as alternative have opposite precedence. See Character Sets and Alternatives (above) for details & examples.

EasyPattern vs. perl regex or grep

At its core, EasyPattern uses "regular expression" technology that is similar to the "regex" or "grep" tools that originated on UNIX. EasyPattern's primary benefit is that the patterns are much easier to read and write.

For those who have some experience with regex, here are a few specific differences:

· Quantity is specified as a prefix rather than a suffix. We believe prefix notation is much more natural.

e.g. [1+ digit] rather than "[0-9]+"

· Parentheses groups are not automatically numbered. Drawback (to some): you have to include a number if you want to refer to that matched portion. Benefits: the parentheses that are there just for logical grouping don't get numbered.
· No backslashes are required to "escape" special characters (instead, EasyPattern provides keywords such as [rightBracket]). Benefit: Other pattern languages already use backslash as an escape character so extra backslashes make patterns even more difficult to read.
· EasyPattern includes keywords for many character sets that require a custom bracketed set in regex, e.g. punctuation, whitespace, paragraph, column, etc.
· EasyPattern keywords generally include Macintosh-specific characters, e.g. [letter] includes letters with umlauts and other diacritical marks
· EasyPattern can combine character sets with [or] (as well as use [or] for alternatives).
· EP's [character] or [char] will match any character; the "equivalent" in some products will match anything except carriage return. If you want any character except a return (or formfeed), use [paragraphChar]; that is, any character that could appear in a paragraph. Of course, [not return] works too.


Status Window

 Status Window
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The Status dialog shows you the progress of the current filtering job - which file is being processed, how many files have been processed, how many files have been skipped (due to Input Filter binary file processing options), the progress in this file, and the overall task progress.

Warnings are shown in Yellow, and Errors in Red. If Errors occur, the title of the tab changes from 'Status' to 'Errors (x)' where x is a running tally.

Double clicking an underlined file will open it. If there is no known association, the file is opened with Notepad. This can be changed using the Preferences dialog.

If the filter was being run in Test Mode (i.e. files with a .dCV output), then an extra button will be displayed after the job has been completed giving you the option to delete all Test files.

If more than 500 lines are logged (this is configurable), file logging to the screen is turned off, and is only logged to file if logging is enabled.

Delete all test files/Restore .BAK files

When Test Mode is enabled, clicking this button removes all test files (.DCV).

When Backup Mode is enabled, clicking this button restores all .BAK files.

Once the files have been deleted/restored, the button is dimmed. These button appear to the left of the Ok/Cancel button only at the end of a filtering job.

Copy lines to log file

This button copies all lines to the log file if they have not already been copied there based on logging settings.

Ok / Cancel

Clicking on the [Cancel] button allows you to stop the job - a confirmation window is displayed. After confirmation, TextPipe may still take several seconds to stop at the next logical break point.

When the job finished, the [Cancel] button changes to an [Ok] button. Clicking the [Ok] button closes the window.

Processing file

The name of the file currently being processed

Binary files skipped

The number of files that have been skipped because they appeared to be binary files.

Non-binary files skipped

The number of files that have been skipped because the user chose to skip them (based on the Input filter settings)

Time elapsed

The amount of time that has elapsed since the filter began processing. This includes time taken waiting for the user to respond to prompting, for example, querying a search and replace.

See also

Drag and Drop

Filter Lists


Wizards

 Wizards
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe has 3 Wizards:

Mainframe Copybook Wizard

Fixed Width to Delimited Wizard

Command line wizard


Mainframe Copybook Wizard

 Mainframe Copybook Wizard
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe can convert mainframe copybook data to CSV (comma-separated), Tab-delimited, Pipe-delimited or fixed-width. Simply paste (or use the right-click menu to load) your copybook data.

See also the Mainframe Copybook Filter (this Wizard just builds a series of filters for you).

Copybook data is accepted in a number of formats - see below.

Use this wizard when you need to see exactly how TextPipe is processing the copybook. This also lets you tweak individual fields if necessary. It will generate a large list of filters. Eventually this wizard will be replaced by the Mainframe copybook filter. This filter does not support XML output yet. If you need the final output in XML, follow this filter with a Convert CSV to XML filter.

TextPipe handles fields and groups with a fixed or variable number of repeats (e.g. OCCURS 8 TIMES. or OCCURS 6 INDEXED BY TN-IDX), but not OCCURS DEPENDING ON FIELDXXX. Use the Mainframe Copybook Filter if you need this.

When you click the [Build] button, the Wizard generates a series of filters:

1. A comment with the entire copybook in it, for future reference
2. Fields are processed in reverse order i.e. right to left. This is because fields may change size (typically get larger) when processed, so this ensures that column positions do not change
3. Each field has a comment that shows the field name, size and column positions, and the text extracted from the copybook. This can be used to cross-check the filters created.
4. Inside the filter comment is a restriction, followed by a filter to expand or convert the data (if not an alpha field). This is followed by an Add Footer filter which outputs the specified field delimiter in EBCDIC form (for CSV, an EBCDIC comma, ASCII k) which then gets converted to an ASCII delimiter later
5. Filters to trim spaces from each field, and to remove the leading + sign from numeric values. These can be removed if you wish.
6. A file header is added with the field names.

If you have an SAS copybook, then first use the filter mainframe\SAS to mainframe copybook.fll to convert it to the format above. If you file has field lengths then data type, use the filter mainframe\convert copybook format.fll to convert it to the correct format.

Output Format

You can choose if TextPipe outputs CSV (comma-separated) data, Tab delimited, Pipe-delimited, Fixed width or XML (filter only). If you use the wizard and need the final output in XML, use the Convert CSV to XML filter.

If TextPipe encounters an invalid packed or zoned decimal digit or sign field, it will output 'x' in the field to indicate an invalid value. If a packed or zoned field contains just spaces, TextPipe will output a blank field.

Assume adjacent COMP fields are overlapped

Odd-length COMP fields have half-bytes (nybbles) of unused data. Some compilers overlap adjacent COMP fields in files to save space. Check this option if your files contains overlapped COMP fields.

Your copybook must show the positions of every packed field and any extra filler COMP fields added by your compiler, as the location of these is system-dependant.

Trim leading spaces and zeroes from the output file

By default, TextPipe leaves leading spaces, zeros and even '+' signs on numerics. Check this option to remove them to save space in the output file.

Allow COMP-3 to be unsigned

Normally, COMP-3 fields have space allocated for a sign, regardless of whether it is used or not.

Check this option if you find that dates are coming out with 7 digits instead of 8. Often a copybook will specify dates as PIC S9(7) COMP-3 instead of PIC 9(8) COMP-3.

This option is supported by the wizard but not yet by the filter.

Only handle numeric fields

When checked, the wizard only generates filters to decode numeric fields.

Remove FILLER fields

When checked, the wizard removes data from fields with a name starting with the word 'FILLER'.

Add field name header

When checked, the wizard adds a file header to the output file.

Sample Formats (1)

TextPipe detects two main COBOL formats:

Both with and without line numbers:

03 TN-INSURANCE-DATA.

05 TN-INS-PRODUCTS OCCURS 6

07 TN-INS-PRODUCT-CODE PIC XX.

07 TN-INS-STATUS PIC X.

07 TN-INS-EFF-DATE PIC 9(8).

07 TN-INS-CANCELLATION-DATE PIC 9(8).

07 TN-INS-REINSTATEMENT-DATE PIC 9(8).

If your code has extra line numbers or other data to the left of the levels (in Red below), please remove it. You can do this in TextPipe by using clicking the mouse at the top left, holding down Alt, and dragging a rectangular block to the bottom right, then press Delete:

05 DD-ACCT-TRSF-REL-IND

PICTURE X.

35766 05 FILLER PICTURE X.

47479 05 DD-ACCT-CMS-ESP PICTURE X.

05 DD-ACCT-BIRTH-DATE PICTURE S9(11) COMPUTATIONAL-3.

If TextPipe does not generate filters correctly, try these things:

1. Remove any extra information from the right hand margin
2. Ensure there is no word wrapping, and ensure every line ends in a period (.)
3. If dates come out with 7 digits instead of 8, check 'Allow COMP-3 to be unsigned'
4. If you have packed fields and the data alignment is not correct, try checking 'Assume adjacent COMP fields are overlapped'
5. If this fails, please send it to us for prompt attention.

Sample Formats (2)

This format without line numbers (the header is optional):

FIELD POSITION

DATA ELEMENT DEFINITION IN RECORD

RECORD CD 1 AN 1

PURCHASE REQUEST NUMBER 14 AN 2-15

PR LINE ITEM NUMBER 6 AN 16-21

QUANTITY 5 P 22-26

UNIT OF ISSUE 2 AN 27-28

STANDARD UNIT PRICE 5 P 29-33

FUND CLASS CODE 3 AN 34-36

STOCK NUMBER 15 AN 37-51


Fixed Width to Delimited Wizard

 Fixed Width to Delimited Wizard
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This wizard saves time by creating a large group of filters to convert fixed width data to CSV (comma-separated), Tab-delimited or Pipe-delimited data. If you need the final output in XML, follow this filter with a Convert CSV to XML filter.

To use this wizard, simply paste (or use the right-click menu to load) your field specification, then click the [Build] button.

Field Specification

TextPipe expects a field specification like this:

Record_Type_P 2

Acct_No_Prefix 2

AcctNo 12

Patient_Status 3

Admit_Dt 8

ABS_Status_Date 8

Patient_Name 30

Pt_Addr_1 30

Pt_Addr_2 30

Pt_City 20

Pt_State 2

Pt_Zip 10

Pt_Sex 1

Pt_DOB 8

Pt_Fin_Class 12

Pt_Phone_No 10

Pt_SSN 9

Pt_Country 10

Pt_County 10

Pt_Discharge_Di 2

Admit_Priority 1

Pt_Weight_in_Kg 7

Leading and trailing spaces are allowed and will be removed.

Output Format

You can choose if TextPipe outputs CSV (comma-separated) data, Tab delimited or Pipe-delimited data. If you need the final output in XML, you can use the Convert CSV to XML filter.

Output Header Record

When checked, TextPipe will add a header record at the top of the file with the field names you have provided. This is very useful for importing the file into Excel or a database.

Trim leading spaces and zeroes from the output file

By default, TextPipe trims leading spaces, zeros and even '+' signs on numerics. Uncheck this option to leave them alone (taking extra space in the output file).


Command line wizard

 Command line wizard
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This function is found in the Tools menu\Command line wizard.

The command line wizard displays actual examples of how to run the current filter from the command line in various modes. Click the Refresh button to display new examples if the current filter changes.

See also

Command line parameters


Input Filter

 Input Filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The Input filter specifies where TextPipe obtains input from.

Input from file(s)

When this radio button is selected, the files given on the "Files to Process" tab will each be used in turn as a source of input data.

Binary files

Generally it isn’t useful to process binary files, because the application they are created by imposes structure on them (unlike text files). If you want all files to be processed (regardless of whether they appear to be binary), choose "Process". If you don’t want any binary files to be processed, choose "Skip". To be given a choice to process a file when TextPipe thinks that it is binary, choose "Prompt before processing".

Note: the Process Inside Compress Files option below requires this setting to be 'Process'.

To determine if a file is binary, TextPipe applies the following rules:

2. If four characters in a row have an ASCII code between 128 and 255, TextPipe immediately assumes the file is binary;
3. If any character read has an ASCII code less than 32 but is not one of the standard formatting codes of 9,10,12,13,26 or 27, TextPipe immediately assumes the file is binary;
4. If less than 80% of characters in the sample are readable, TextPipe assumes the file is binary. The number of characters in the sample can be configured using the Sample Size field.

This option can be enabled on the command line by using the /B=P (process binary files) or /B=S (skip binary files) option. The option to prompt cannot be chosen from the command line.

Sample Size

If you find that your binary files are not being identified correctly, you can increase the sample size of text that TextPipe examines to determine if a file is binary.

Confirm before processing each file

TextPipe will ask the user to confirm processing each of read only file that it encounters.

This option cannot be enabled from the command line.

Confirm before processing read only files

Before processing each file, TextPipe prompts the user to confirm that the file should be processed.

This option cannot be enabled from the command line.

Delete input files after processing

This option deletes the input files after they have been processed. It is useful when the original files are not required after transformation, but should be used with EXTREME CAUTION.

This option cannot be enabled from the command line.

Process inside compressed files (ZIP, DOCX, XLSX and PPTX, ODT, ODS, ODP, XPS)

This option allows TextPipe to process the compressed files inside a .ZIP, Microsoft 2007 Word DOCX, Excel .XLSX and PowerPoint PPTX files, and OpenOffice Writer ODT, Calc ODS and Impress ODP files.

TextPipe can also identify PKZIP compressed files of unknown file type and process them accordingly.

All files are extracted to a folder structure inside My Documents\DataMystic\TextPipe\tempzip\.. Any modified files are recompressed and returned to the compressed file, the remaining files are deleted. The Restrict to Files Matching Internal Zip Filename filter can be used to control which files inside the Zip file a set of filters is applied to.

This option cannot be enabled from the command line. The Process Binary Files option above MUST be set to 'Process' for this option to work.

Input from clipboard

When this radio button is selected, the current contents of the Windows clipboard is used as the source of input data.

Unicode Input (UTF16-LE)

When true, input from the clipboard arrives in Unicode format (required for international character sets), otherwise ANSI is used.

This option cannot be enabled from the command line.

Input from Trial Run Input

When this radio button is selected, TextPipe uses the contents of the Trial Run Input Area for its input. This is very handy to avoid having to save a text file to provide the input.

Generate Random Input of Size ...

This filter is used to generate files of a specific size, containing purely random data. The resulting file can be used as a test file for other applications. The data is generated by a pseudo-random number generator, and is different each time the filter is run. The characters generated are in the full ASCII range of 0 to 255 and have a normal distribution. The maximum file size than can be generated is 2147483647 bytes (2 GB).

See also

Output filter


Output Filter

 Output Filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This filter specifies where TextPipe sends output to. You can also use a Secondary Output Filter to send data to alternate locations.

Output to file(s)

Output Mode

This option controls how the original files are preserved.

Normal mode - the original files are overwritten with the processed results. This is the default.

Test mode - processed files are output to a .dCv extension

The processed file is the original file's name with '.dCv' added to it. This is handy to ensure that you don’t destroy your original file. You can quickly delete the test files by using the [Delete Test Files] button in the results dialog before closing it. This option can be enabled on the command line by using the /T+ option.

Backup mode - The original files are renamed by adding a .bak extension

The original file has a .bak extension added (to prevent conflict with files like common.html/common.php having the same .bak file), and the processed file is output to the original filename. This is handy to ensure that you don’t destroy your original file, and this option is highly recommended. You can quickly restore the original files by using the [Restore .BAK Files] button in the results dialog before closing it. This option can be enabled on the command line by using the /TB option.

Only output modified files

When this option is selected, only files that are changed by the filter list will be marked as changed with a new date. This makes it easier to find which files have been affected or not.

If the output filename is different to the input filename then this setting is ignored - the file is always written to disk. If you want to know which files have changed, use the Keep original file’s date/time option below.

Keep original file’s date/time

The processed file is given the same date and time as the original file.

This option can be enabled on the command line by using the /N option. Retaining the date and time is disabled by default.

Append to existing file

When checked, this setting will append output to the existing output file rather than first deleting the existing output file.

Change extension to

Changes the extension of the output file to the given value when selected. The extension given can optionally include a leading period '.', one will be added if it does not have one. This option is useful when you are converting from one file type to another e.g. HTML to .txt, .pas to .c etc. This field can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt, and macros or global variables. TextPipe corrects any doubled backslashes.

Output folder

Changes the folder that the output files are stored in. This is useful if you want to keep the new files completely separate from the old files, or if you do not have permission to write to the folder where the original files are kept.

TextPipe will create the destination folder if it does not exist.

A backslash '\' is added to the folder name if it does not have one. This field can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt, and macros or global variables. TextPipe corrects any doubled backslashes.

Maintain folder structure

If a new output folder is selected, all files will be output into the specified folder regardless of their original folder. If the Maintain folder structure check box is selected, the original folder structure is maintained beneath the starting directory. TextPipe will create the destination folder if it does not exist.

Ignore Initial Folder

Ignores the start folder name when generating the target folder name.

Remove empty output files

If an output file is empty (ie has a size of zero bytes) then checking this option will remove it.

Output to clipboard (join input files together)

When this radio button is selected, all output is joined together and sent to the Windows clipboard.

Note: This clipboard only sends data to the clipboard at the end of a processing job - not once for each file.

Unicode Input (UTF16-LE)

When true, output is sent to the clipboard in Unicode format (required for international character sets), otherwise ANSI is used.

This option cannot be enabled from the command line.

Single file Output (join input files together)

When this radio button is selected, all output is joined together and sent to the specified file. If you need to join files together and then perform further text processing, such as adding headers, footers, sorting and/or removing duplicates, then use a separate Merge Filter instead of using this option. Set the Output Filter to File(s) Output mode.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt, and macros or global variables. TextPipe corrects any doubled backslashes.

See also

Input filter


Conversion Filters

 Conversion Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert

These filters generally do not delete or add information, but change it into a different form.

Convert End of Line Characters

Convert IBM Drawing Characters

Tabs and spaces

Tabs to Spaces

Spaces to Tabs

Character Case

UPPERCASE

lowercase

Title Case

Sentence case.

tOGGLE cASE

raNDoM cASe

Code page conversions

Numeric HTML entities to text (Standard and Pro)

UTF16-LE to Escaped Unicode (ASCII + UCN \U) (Standard and Pro)

Character set

ASCII/OEM to ANSI (Pro version only)

ANSI to ASCII/OEM (Pro version only)

Mainframe - see also the mainframe copybook wizard

Fixed width - see the Fixed Width to Delimited Wizard

ASCII to EBCDIC (Pro version only)

EBCDIC to ASCII (Pro version only)

Expand packed decimal (Pro version only)

Expand unsigned odd/even length packed numeric

Expand packed numeric (use the Map Expand packed numeric.map, and Expand EBCDIC BCD.map)

Expand zoned decimal (Pro version only)

Compress packed decimal (Pro version only)

Compress zoned decimal (Pro version only)

Expand Binary Number to ASCII/EBCDIC (Pro version only)

Records (same as Convert end of line characters)

Unicode - now found under the Unicode Menu

CSV

CSV to Tab-delimited (Pro version only)

CSV to XML (Pro version only)

Tab-delimited

Tab-delimited to CSV (Pro version only)

Tab-delimited to XML (Pro version only)

Text to Word List (Pro version only)

Word documents to text files (Pro version only)

Excel documents to text files (Pro version only)

PDF documents to text (Pro version only)

Hex encoding and Hex decoding

HTTP Encode and HTTP Decode

Mime Base 64 encode and Mime Base 64 decode

Mime quoted printable encode and Mime quoted printable decode

UUEncode and UUDecode

XXEncode and XXDecode

Unscramble

Hex dump (Pro version only)

Decimal dump (Pro version only)

Character maths (Pro version only)


End of Line characters

 End of Line characters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert

The Operating Systems used by various computers do not share a standard symbol that represents the end of each line in text files. Unix and the Amiga both use ASCII code 10, Macintoshes use ASCII code 13, and PCs use ASCII code 13 followed by 10.

This difference introduces an extra complication into transferring files between computers – the end of line characters must be converted to match the target system. Mainframes do not use end of line characters, but instead assume a fixed length for each line.

Input line feeds

Select the line feed type of the input files.

Auto

If you are unsure of the source system, select the "Auto" option. The "Auto" option can reliably detect and correct text files containing a variety of line feeds.

Fixed length

If you are converting a mainframe file that contains fixed length records, select "Fixed length" and enter the record length. The maximum record length is 2147483647.

Note: If you are converting 132 column mainframe reports, you should set the fixed length to 133, because each line has a prefix character.

Mainframe CMS Format

This setting converts variable length EBCDIC data to ASCII delimited lines, each terminated by the selected Output line feed character.

Remove bad EOLs

If you suspect that the input file contains different end of line symbols than it should (eg a Unix text file containing ASCII code 13s), then check "Remove Bad EOLs". Note that this option can be dangerous – because if the file has no line feeds of the type that you select, all line feeds will be removed. Always use a test run first to make sure.

Output line feeds

Select the line feed type to output. If you want all line feeds to be removed, select "None". If you want to specify your own character or group of characters, select the radio button beside the edit field and enter the text you wish to be used. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters.


IBM Drawing Characters

 IBM Drawing Characters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

IBM drawing characters in the upper ASCII range (128-255) are commonly used to draw lines and boxes, single and double line borders, shaded characters etc. Many devices (eg printers, non-IBM computers etc) do not support the display of these characters.

This filter converts them to standard ASCII characters (+, - and |) that all computers can display.


Tabs to Spaces

 Tabs to Spaces
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Expands tabs out into the equivalent number of spaces, usually 8. This filter is used to make files easier to process with programs that support differing tab sizes.

Some text editors expect a specific tab size, and save files with the tabs embedded. To import these files successfully into text editors that use a different tab size, you need to use this filter to first convert the tabs to spaces.

This filter may increase the size of a file.

See also:

· Spaces to Tabs


Spaces to Tabs

 Spaces to Tabs
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Reduces spaces into the appropriate number of tabs. There are usually 8 spaces per tab. This filter is used to make files smaller.

See also:

· Tabs to Spaces


UPPERCASE

 UPPERCASE
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Forces all text to UPPERCASE. To make the conversion, the function uses the current language selected by the user in the system Control Panel. If no language has been selected, TextPipe uses the Windows internal default mapping. The conversion is made based on the code page associated with the process locale.

If your input data is Unicode, check the Unicode (UTF-16LE) check box. You must also ensure that the Unicode input is in UTF-16LE format. Use the Convert to/from Unicode filter to convert it if not.

See also:

lowercase

Title Case

Sentence case

tOGGLE cASE

raNDoM cASe


lowercase

 lowercase
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Forces all text to lowercase. To make the conversion, the function uses the current language selected by the user in the system Control Panel. If no language has been selected, TextPipe uses the Windows internal default mapping. The conversion is made based on the code page associated with the process locale.

If your input data is Unicode, check the Unicode (UTF-16LE) check box. You must also ensure that the Unicode input is in UTF-16LE format. Use the Convert to/from Unicode filter to convert it if not.

See also:

UPPERCASE

Title Case

Sentence case

tOGGLE cASE

raNDoM cASe


Title Case

 Title Case
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts all text to Title Case ie the first letter of every word is capitalised, and all other letters are forced to lower case. This routine calculates a table of upper and lower case letters on TextPipe startup, and this determination is based on the semantics of the language selected in Control Panel.

If your input data is Unicode, check the Unicode (UTF-16LE) check box. You must also ensure that the Unicode input is in UTF-16LE format. Use the Convert to/from Unicode filter to convert it if not.

See also:

UPPERCASE

lowercase

Sentence case

tOGGLE cASE

raNDoM cASe


Sentence case

 Sentence case
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts all text to Sentence case ie the first word in every sentence is capitalised, all other letters are left as is. Sentences are deemed to start after full stops, exclamation marks, colons, question marks, quotes, brackets and angle brackets (.!:?'"<().

This routine calculates a table of upper and lower case letters on TextPipe startup, and this determination is based on the semantics of the language selected in Control Panel.

See also:

UPPERCASE

lowercase

Title Case

tOGGLE cASE

raNDoM cASe


tOGGLE cASE

 tOGGLE cASE
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

tOGGLES tHE cASE of all text ie all UPPERCASE characters are converted to lowercase and vice-versa.

This routine calculates a table of upper and lower case letters on TextPipe startup, and this determination is based on the semantics of the language selected in Control Panel.

See also:

UPPERCASE

lowercase

Title Case

Sentence case

raNDoM cASe


raNDoM cASe

 raNDoM cASe
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This filter randomly changes the case of characters.

This routine calculates a table of upper and lower case letters on TextPipe startup, and this determination is based on the semantics of the language selected in Control Panel.

Running this filter again will generate different results e.g.

1. ranDoMIze cASe
2. RanDOmIZE case
3. randOMIZE casE

See also:

UPPERCASE

lowercase

Title Case

Sentence case

tOGGLE cASE


Code page conversion

 Code page conversion
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro)

The Unicode conversion filter offers conversions between the following code pages:

For more information on code pages, please see

· Microsoft Locale-Specific Code Page Information
· Character Sets And Code Pages At The Push Of A Button

Code-Page Identifiers

(*) The list of available code pages may be different on your system. You can install additional code pages using Control Panel\Regional Options.

Identifier | Name
037 | IBM EBCDIC - U.S./Canada
437 | OEM - United States
500 | IBM EBCDIC - International
708 | Arabic - ASMO 708
709 | Arabic - ASMO 449+, BCON V4
710 | Arabic - Transparent Arabic
720 | Arabic - Transparent ASMO
737 | OEM - Greek (formerly 437G)
775 | OEM - Baltic
850 | OEM - Multilingual Latin I
852 | OEM - Latin II
855 | OEM - Cyrillic (primarily Russian)
857 | OEM - Turkish
858 | OEM - Multlingual Latin I + Euro symbol
860 | OEM - Portuguese
861 | OEM - Icelandic
862 | OEM - Hebrew
863 | OEM - Canadian-French
864 | OEM - Arabic
865 | OEM - Nordic
866 | OEM - Russian
869 | OEM - Modern Greek
870 | IBM EBCDIC - Multilingual/ROECE (Latin-2)
874 | ANSI/OEM - Thai (same as 28605, ISO 8859-15)
875 | IBM EBCDIC - Modern Greek
932 | ANSI/OEM - Japanese, Shift-JIS
936 | ANSI/OEM - Simplified Chinese (PRC, Singapore)
949 | ANSI/OEM - Korean (Unified Hangeul Code)
950 | ANSI/OEM - Traditional Chinese (Taiwan; Hong Kong SAR, PRC)
1026 | IBM EBCDIC - Turkish (Latin-5)
1047 | IBM EBCDIC - Latin 1/Open System
1140 | IBM EBCDIC - U.S./Canada (037 + Euro symbol)
1141 | IBM EBCDIC - Germany (20273 + Euro symbol)
1142 | IBM EBCDIC - Denmark/Norway (20277 + Euro symbol)
1143 | IBM EBCDIC - Finland/Sweden (20278 + Euro symbol)
1144 | IBM EBCDIC - Italy (20280 + Euro symbol)
1145 | IBM EBCDIC - Latin America/Spain (20284 + Euro symbol)
1146 | IBM EBCDIC - United Kingdom (20285 + Euro symbol)
1147 | IBM EBCDIC - France (20297 + Euro symbol)
1148 | IBM EBCDIC - International (500 + Euro symbol)
1149 | IBM EBCDIC - Icelandic (20871 + Euro symbol)
1200 | Unicode UCS-2 Little-Endian (BMP of ISO 10646)
1201 | Unicode UCS-2 Big-Endian
1250 | ANSI - Central European
1251 | ANSI - Cyrillic
1252 | ANSI - Latin I
1253 | ANSI - Greek
1254 | ANSI - Turkish
1255 | ANSI - Hebrew
1256 | ANSI - Arabic
1257 | ANSI - Baltic
1258 | ANSI/OEM - Vietnamese
1361 | Korean (Johab)
10000 | MAC - Roman
10001 | MAC - Japanese
10002 | MAC - Traditional Chinese (Big5)
10003 | MAC - Korean
10004 | MAC - Arabic
10005 | MAC - Hebrew
10006 | MAC - Greek I
10007 | MAC - Cyrillic
10008 | MAC - Simplified Chinese (GB 2312)
10010 | MAC - Romania
10017 | MAC - Ukraine
10021 | MAC - Thai
10029 | MAC - Latin II
10079 | MAC - Icelandic
10081 | MAC - Turkish
10082 | MAC - Croatia
12000 | Unicode UCS-4 Little-Endian
12001 | Unicode UCS-4 Big-Endian
20000 | CNS - Taiwan
20001 | TCA - Taiwan
20002 | Eten - Taiwan
20003 | IBM5550 - Taiwan
20004 | TeleText - Taiwan
20005 | Wang - Taiwan
20105 | IA5 IRV International Alphabet No. 5 (7-bit)
20106 | IA5 German (7-bit)
20107 | IA5 Swedish (7-bit)
20108 | IA5 Norwegian (7-bit)
20127 | US-ASCII (7-bit)
20261 | T.61
20269 | ISO 6937 Non-Spacing Accent
20273 | IBM EBCDIC - Germany
20277 | IBM EBCDIC - Denmark/Norway
20278 | IBM EBCDIC - Finland/Sweden
20280 | IBM EBCDIC - Italy
20284 | IBM EBCDIC - Latin America/Spain
20285 | IBM EBCDIC - United Kingdom
20290 | IBM EBCDIC - Japanese Katakana Extended
20297 | IBM EBCDIC - France
20420 | IBM EBCDIC - Arabic
20423 | IBM EBCDIC - Greek
20424 | IBM EBCDIC - Hebrew
20833 | IBM EBCDIC - Korean Extended
20838 | IBM EBCDIC - Thai
20866 | Russian - KOI8-R
20871 | IBM EBCDIC - Icelandic
20880 | IBM EBCDIC - Cyrillic (Russian)
20905 | IBM EBCDIC - Turkish
20924 | IBM EBCDIC - Latin-1/Open System (1047 + Euro symbol)
20932 | JIS X 0208-1990 & 0121-1990
20936 | Simplified Chinese (GB2312)
21025 | IBM EBCDIC - Cyrillic (Serbian, Bulgarian)
21027 | Extended Alpha Lowercase
21866 | Ukrainian (KOI8-U)
28591 | ISO 8859-1 Latin I
28592 | ISO 8859-2 Central Europe
28593 | ISO 8859-3 Latin 3
28594 | ISO 8859-4 Baltic
28595 | ISO 8859-5 Cyrillic
28596 | ISO 8859-6 Arabic
28597 | ISO 8859-7 Greek
28598 | ISO 8859-8 Hebrew
28599 | ISO 8859-9 Latin 5
28605 | ISO 8859-15 Latin 9
29001 | Europa 3
38598 | ISO 8859-8 Hebrew
50220 | ISO 2022 Japanese with no halfwidth Katakana
50221 | ISO 2022 Japanese with halfwidth Katakana
50222 | ISO 2022 Japanese JIS X 0201-1989
50225 | ISO 2022 Korean
50227 | ISO 2022 Simplified Chinese
50229 | ISO 2022 Traditional Chinese
50930 | Japanese (Katakana) Extended
50931 | US/Canada and Japanese
50933 | Korean Extended and Korean
50935 | Simplified Chinese Extended and Simplified Chinese
50936 | Simplified Chinese
50937 | US/Canada and Traditional Chinese
50939 | Japanese (Latin) Extended and Japanese
51932 | EUC - Japanese
51936 | EUC - Simplified Chinese
51949 | EUC - Korean
51950 | EUC - Traditional Chinese
52936 | HZ-GB2312 Simplified Chinese
54936 | Windows XP: GB18030 Simplified Chinese (4 Byte)
57002 | ISCII Devanagari
57003 | ISCII Bengali
57004 | ISCII Tamil
57005 | ISCII Telugu
57006 | ISCII Assamese
57007 | ISCII Oriya
57008 | ISCII Kannada
57009 | ISCII Malayalam
57010 | ISCII Gujarati
57011 | ISCII Punjabi
65000 | Unicode UTF-7
65001 | Unicode UTF-8


Convert Numeric HTML/XML entities to text

 Convert Numeric HTML/XML entities to text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro)

This filter converts decimal/hex numeric HTML/XML entities to plain text e.g.

&#174; -> ®

&#xAE; -> ®

Typically, the input file is ANSI (single byte) format. This filter will output UTF-8 characters for high-value entities e.g. &#6144; The best approach is to first convert the file from ANSI to UTF-8, then apply this filter.

To convert plain text to HTML entities (ie convert in the opposite direction), use a Map or Unicode Map.


Convert HTML/XML entities to text

 Convert HTML/XML entities to text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro)

This filter converts decimal/hex numeric HTML/XML entities and entity names to plain text e.g.

&#174; -> ®

&#xAE; -> ®

&reg; -> ®

Typically, the input file is ANSI (single byte) format. This filter will output UTF-8 characters for high-value entities e.g. &#6144; The best approach is to first convert the file from ANSI to UTF-8, then apply this filter.

To convert plain text to HTML entities (ie convert in the opposite direction), use a Map or Unicode Map.


ASCII/OEM to ANSI

 ASCII/OEM to ANSI
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro)

Converts from ASCII/OEM to ANSI. ANSI is the character set used by Windows, and it includes all accentuated Roman characters used by non-English languages like French, German and Spanish. ASCII/OEM is an extension of the original IBM character set where various non-essential characters are replaced by language-specific accentuated characters. Different ASCII/OEM character sets are not compatible. They must be converted to ANSI and then back to the correct ASCII/OEM character set to be readable.

This filter can be useful if you want to send a text file to someone using a language other than your own.

Also known as a DOS to ANSI filter.

See also:

ANSI to ASCII/OEM


ANSI to ASCII/OEM

 ANSI to ASCII/OEM
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro)

Converts from ANSI to ASCII/OEM. ANSI is the character set used by Windows, and it includes all accentuated Roman characters used by non-English languages like French, German and Spanish. ASCII/OEM is an extension of the original IBM character set where various non-essential characters are replaced by language-specific accentuated characters. Different ASCII/OEM character sets are not compatible. They must be converted to ANSI and then back to the correct ASCII/OEM character set to be readable.

This filter can be useful if you want to send a text file to someone using a language other than your own.

Also known as an ANSI to DOS filter.

See also:

ASCII/OEM to ANSI


CSV to Tab-delimited

 CSV to Tab-delimited
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts CSV data (quoted or unquoted) to tab-delimited form. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. If the data is properly quoted then TextPipe will determine this automatically.

e.g. Input CSV file:

1,,3,,5,,7,,9,,11,,13,

"1","2","3","4","5","6","7","8","9","10","11","12","13","14"

1,2,3,4,5,6,7,8,9,10,11,12,13,14

1,2,3,4,5,6,7,8,9,10,11,12,13,

"1","2","3","4","53424""34242","""3243246","7","8","9","10","11","12","13","14"

Output Tab-delimited file (Tabs are not shown correctly)

1 3 5 7 9 11 13

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13

1 2 3 4 53424"34242 "3243246 7 8 9 10 11 12 13 14


CSV to XML

 CSV to XML
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts CSV data (quoted or unquoted) to XML form. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. If the data is properly quoted then TextPipe will determine this automatically.

TextPipe correctly escapes < > " ' and & in the data to the corresponding XML entity. If your data contains invalid XML characters such as ASCII 26 (End-of-file, hex \x1A), follow this filter with a search/replace filter to remove \x1A and replace with nothing.

If your input file co

e.g. Input CSV file:

1,,3,,5,,7,,9,,11,,13,

"1","2","3","4","5","6","7","8","9","10","11","12","13","14"

1,2,3,4,5,6,7,8,9,10,11,12,13,14

1,2,3,4,5,6,7,8,9,10,11,12,13,

"1","2","3","4","53424""34242","""3243246","7","8","9","10","11","12","13","14"

Output XML format

<line>

<field1>1</field1>

<field2></field2>

<field3>3</field3>

<field4></field4>

<field5>5</field5>

<field6></field6>

<field7>7</field7>

<field8></field8>

<field9>9</field9>

<field10></field10>

<field11>11</field11>

<field12></field12>

<field13>13</field13>

<field14></field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>5</field5>

<field6>6</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14>14</field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>5</field5>

<field6>6</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14>14</field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>5</field5>

<field6>6</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14></field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>53424"34242</field5>

<field6>"3243246</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14>14</field14>

</line>


Tab-delimited to CSV

 Tab-delimited to CSV
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts Tab-delimited data to CSV data. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. TextPipe cannot determine this without column headers.

Input Tab-delimited file (Tabs are not shown correctly)

1 3 5 7 9 11 13


Tab-delimited to XML

 Tab-delimited to XML
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts Tab-delimited data to XML data. It's preferable to use a file with column headers, because then TextPipe can easily determine if the fields have embedded CR/LFs in them. TextPipe cannot determine this without column headers.

TextPipe correctly escapes < > " ' and & in the data to the corresponding XML entity. If your data contains invalid XML characters such as ASCII 26 (End-of-file, hex \x1A), follow this filter with a search/replace filter to remove \x1A and replace with nothing.

Input Tab-delimited file (Tabs are not shown correctly)

1 3 5 7 9 11 13

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13 14

1 2 3 4 5 6 7 8 9 10 11 12 13

1 2 3 4 53424"34242 "3243246 7 8 9 10 11 12 13 14

Output XML format

<line>

<field1>1</field1>

<field2></field2>

<field3>3</field3>

<field4></field4>

<field5>5</field5>

<field6></field6>

<field7>7</field7>

<field8></field8>

<field9>9</field9>

<field10></field10>

<field11>11</field11>

<field12></field12>

<field13>13</field13>

<field14></field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>5</field5>

<field6>6</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14>14</field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>5</field5>

<field6>6</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14>14</field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>5</field5>

<field6>6</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14></field14>

</line>

<line>

<field1>1</field1>

<field2>2</field2>

<field3>3</field3>

<field4>4</field4>

<field5>53424"34242</field5>

<field6>"3243246</field6>

<field7>7</field7>

<field8>8</field8>

<field9>9</field9>

<field10>10</field10>

<field11>11</field11>

<field12>12</field12>

<field13>13</field13>

<field14>14</field14>

</line>


Text to Word List

 Text to Word List
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This filter takes all the incoming words and outputs them one per line, with a DOS line feed between them. This can be used to generate word lists for Indexes, encryption programs etc. Hyphenated words are recognised as single words, provided that they aren't broken across lines. To get around this limitation, use a Search and Replace filter to replace hyphens followed by line feeds with just a hyphen.

Normally you would follow this filter with a Sort and Remove Duplicates filter, or alternatively, a Count Duplicate Lines filter (with Include counts of 1).

catch22 - a word

24-7 - a word

twenty-four - a word

5th - a word

ice cream - two words

Commas or periods after words are treated as word separators.


Word documents to text

 Word documents to text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro version)

This filter takes ALL incoming documents, opens them with Microsoft Word, and outputs them as text files. This can be used to process a set of Word Documents to text file format. After this filter you can add search and replace filters or any other filters you choose.

This filter sets the output extension to .TXT.

This filter requires Microsoft Word 98 or higher to be installed. If you wish to convert documents other than the default .DOC files, you may also need to install Word's conversion filters. If Word cannot be started automatically TextPipe will prompt you to start it manually before continuing.

Unless you know that all documents being processed are Word documents (e.g. by using a wildcard of *.doc in the Files to Process tab), you should restrict this filter to only files matching the pattern

\.DOC$

See also

Word documents to text

Excel spreadsheets to CSV

PDF documents to text


Excel spreadsheets to CSV

 Excel spreadsheets to CSV
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro version)

This filter takes ALL incoming documents, opens them with Microsoft Excel, and outputs them as CSV (comma-delimited) files. This can be used to process a set of Excel Spreadsheets to CSV. After this filter you can add search and replace filters or any other filters you choose, such as convert the data to Tab-delimited or XML.

This filter sets the output extension to .CSV.

This filter requires Microsoft Excel 98 or higher to be installed. If you wish to convert documents other than the default .XLS files, you may also need to install Excel's conversion filters. If Excel cannot be started automatically TextPipe will prompt you to start it manually before continuing.

TextPipe adds a form feed between each sheet in the output file.

As of TextPipe 9.7 and above, this filter skips hidden (and very hidden) sheets.

Unless you know that all documents being processed are Excel documents (e.g. by using a wildcard of *.xls in the Files to Process tab), you should restrict this filter to only files matching the pattern

\.XLS$

See also

Word documents to text

Excel spreadsheets to CSV

PDF documents to text


Convert PDFs to text

 Convert PDFs to text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro version)

This filter takes ALL incoming documents and converts them from PDF to text. Most formatting will be lost. After this filter you can add search and replace filters or any other filters you choose.

This filter sets the output extension to .TXT.

Unless you know that all documents being processed are PDFs (e.g. by using a wildcard of *.pdf in the Files to Process tab), you should restrict this filter to only files matching the pattern

\.PDF$

See also

Word documents to text

Excel spreadsheets to CSV

PDF documents to text


Word concordance
 Word concordance
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Standard and Pro version only)

This filter generates a word concordance. A word concordance shows the context or surrounding words for a given set of words in a dictionary.

e.g. for the line above, a word concordance for the single-word dictionary 'a'

a 1 generates a word

a 1 concordance. A word

a 1 for a given

a 1 in a dictionary

The concordance also gives the line number and the actual word.

Before

The number of words to include prior to the matching word.

After

The number of words to include after the matching word.

Output Format

How the output string is formatted. The order is

0. The dictionary word that was found (string or %s)
1. The line number (integer or %d)
2. The context (string or %s)

You can change the output format using Format Specifiers. e.g.

· %1:d %2:s - Ignores the found dictionary word, and just displays the line number and context
· %0:10.10s %1:5.5d %2:s - Shows 5 digit line numbers with leading zeroes
· %-10.10s %5.5d %s - Shows words right justified in 10 columns.

Dictionary of words to find

The list of words to report on. If you need a report on ALL words, you can use Filters\Convert\Text to word list to generate a full word list.

Note that the dictionary is sorted and all duplicates removed, so you don't need to do this yourself.


Hex encoding

 Hex encoding
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

A very simple encoding of a file. Usually used for small files, because it wastes a large amount of space. The benefit is that the file is very easy to encode/decode, and the file cannot be corrupted passing through mail gateways.

Text that has been processed with the Hex encoding filter usually looks like the following:

496620796F7520626F74686572656420746F206465636F646520746869732C20796F75207265

616C6C79206861766520746F6F206D7563682074696D65206F6E0D0A796F75722068616E6473

21

See also

Hex decoding


Hex decoding

 Hex decoding
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Converts a file from its hex representation back to binary.

The file to be decoded MUST NOT have any extra characters at the start or end if it is to be successfully processed.

See also

Hex encoding


HTTP Encode

 HTTP Encode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This filter is used to encode text for use in an HTTP header – a (usually) small piece of text that accompanies a web page request to a web server. This filter is very useful for debugging CGI scripts because it can create HTTP requests in the correct form. HTTP encoded text usually looks like the following:

a+%28usually%29+small+piece+of+text+that+accompanies+a+web+page+request+to+a+web+server.+This+filter+is+very+

See also

HTTP Decode


HTTP Decode

 HTTP Decode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This filter is used to decode text from an HTTP header – a (usually) small piece of text that accompanies a web page request to a web server.

See also

HTTP Encode


Mime Base 64 encode

 Mime Base 64 encode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Used for binary data. Files of this type are usually given an extension of .b64.

Text that has been processed with the Mime Base 64 filter usually looks like the following:

SWYgeW91IGJvdGhlcmVkIHRvIGRlY29kZSB0aGlzLCB5b3UgcmVhbGx5IGhhdmUgdG9vIG11Y2gg

dGltZSBvbg0KeW91ciBoYW5kcyENCg==

See also

Mime quoted printable encode

Mime quoted printable decode

Mime Base 64 decode


Mime Base 64 decode

 Mime Base 64 decode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Used for binary data. Files of this type are usually given an extension of .b64.

The file to be decoded MUST NOT have any extra characters at the start or end if it is to be successfully processed.

See also

Mime quoted printable encode

Mime quoted printable decode

Mime Base 64 encode


Mime quoted printable encode

 Mime quoted printable encode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Quoted printable is used for text that is mainly readable, but may contain special characters with accents etc.

See also

Mime quoted printable decode

Mime Base 64 encode

Mime Base 64 decode


Mime quoted printable decode

 Mime quoted printable decode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

The inverse of the above encoding.

See also

Mime quoted printable encode

Mime Base 64 encode

Mime Base 64 decode


UUEncode

 UUEncode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Usually used for transmitting binary files inside an email. Files of this type are usually given an extension of .uue. Warning – UUencoded text may be corrupted when passing over a mainframe mail gateway. To avoid corruption, use Mime Base 64 or XXEncode.

UUEncoding's major drawback is that there are many flavors of uuencode in existence and this plays against interoperability. Gateways that convert between EBCDIC and ASCII, furthermore, may damage some of the characters used in the uuencode format. The base64 encoding is designed to be invulnerable to all known gateways. The UUencoding implemented by TextPipe uses the following characters:

`!#$%&''()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_

+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Text that has been processed with the UUEncode filter usually looks like the following:

UUEncoded attachment by TextPipe Evaluation Edition

begin 0777 clipboard

M268@>6]U(&)O=&AE<F5D('1O(&1E8V]D92!T:&ES+"!Y;W4@<F5A;&QY(&AA

B=F4@=&]O(&UU8V@@=&EM92!O;@T*>6]U<B!H86YD<R$-"@``

`

end

See also

UUDecode

XXEncode

XXDecode


UUDecode

 UUDecode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Mail attachments can be uuencoded, use this filter to convert the file back to its correct form. Files of this type are usually given an extension of .uue.

UUEncoding's major drawback is that there are many flavors of uuencode in existence and this plays against interoperability. Gateways that convert between EBCDIC and ASCII, furthermore, may damage some of the characters used in the uuencode format. The base64 encoding is designed to be invulnerable to all known gateways. The UUencoding implemented by TextPipe uses the following characters:

`!#$%&''()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_

See also

UUEncode

XXEncode

XXDecode


XXEncode

 XXEncode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Essentially identical to UUEncode except that the character set used is different to allow it to pass through EBCDIC gateways without corruption.

The XXencoding implemented by TextPipe uses the following characters:

+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Text that has been processed with the XXEncode filter usually looks like the following:

XXEncoded attachment by TextPipe Evaluation Edition

begin 0777 clipboard

hGKMUSKxp647jR4VZQaJY65Fj64FZMqxYNG-oO4Zn90-tPrIUQaJVP4lt64VV

WRaIUR4xj64ppMqUUR4ZhNG-jPUo8SKxpQW-cMKtYQm2B0U++

+

end

See also

UUEncode

UUDecode

XXDecode


XXDecode

 XXDecode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Essentially identical to UUDecode except that the character set used is different to allow it to pass through EBCDIC gateways without corruption.

The XXencoding implemented by TextPipe uses the following characters:

+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

See also

UUEncode

UUDecode

XXEncode


Unscramble (ROT13)

 Unscramble (ROT13)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This is a simple email encoding usually used to disguise text that some people may find offensive. The encoding is totally reversible (applying it twice removes the encoding). Only alpha characters are affected (A..Z and a..z).

Text that has been processed with the ROT13 filter usually looks like the following:

Vs lbh obgurerq gb qrpbqr guvf, lbh ernyyl unir gbb zhpu gvzr ba

lbhe unaqf!


Hex dump

 Hex dump
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This changes the text to lines consisting of 16 bytes each. Each line has an 8 hex digit file index, 16 bytes (in hex) and the ASCII representation:

00000000 65 67 69 6E 0D 0A 20 20 20 20 20 20 61 64 64 72 egin.. addr

00000010 65 73 73 20 3A 3D 20 0D 0A 20 20 20 20 20 20 20 ess := ..

00000020 20 64 65 63 54 6F 48 65 78 53 74 72 28 20 28 66 decToHexStr( (f

This filter is very useful for identifying special characters to search and replace.

If you need to simply convert a file to hex, use the Email\Hex encoding filter.


Decimal dump

 Decimal dump
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This changes the text to lines consisting of 10 bytes each. Each line has a 10 decimal digit file index, 10 bytes (in decimal) and the ASCII representation:

0000000000 080 108 101 097 115 101 032 102 101 101 Please fee

0000000010 108 032 102 114 101 101 032 116 111 032 l free to

0000000020 099 111 109 109 101 110 116 013 010 111 comment..o

This filter is very useful for identifying special characters to search and replace.

If you need to simply convert a file to hex, use the Email\Hex encoding filter.


Character maths

 Character maths
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

Performs a mathematical translation on the input characters. All operations are performed modulo 256.

The operations supported are:

+

-

*

div (the remainder is ignored)

mod (the remainder after division)

xor

and

or

not

shift left (0 inserted)

shift right (0 inserted)

rotate left

rotate right

The hex and binary equivalent of the operand is displayed next to the decimal value that can be edited.

Operand

The maximum operand value is 255.


Calculation

 Calculation
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Convert (Std and Pro version only)

This filter is designed to be run inside a restriction, such as inside a Restrict to CSV/Tab filter, to only operate on a small range of numerical text. If you use a Restrict to CSV/Tab filter, ensure that it is set to remove delimiters and/or quotes.

Performs a mathematical calculation on the input text - which must be numeric.

The operations supported are:

+

-

*

div (the remainder is ignored)

mod (the remainder after division)

shift left (multiply by 2)

shift right (divide by 2)

Operand

The operand can be any number (not just decimal).


Mainframe

 Mainframe
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)


Mainframe Copybook Filter

 Mainframe Copybook Filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

(Pro version only)

TextPipe can convert mainframe copybook data to CSV (comma-separated), Tab-delimited, Pipe-delimited, fixed-width or XML. Simply paste (or use the right-click menu to load) your copybook data.

Copybook data is accepted in a number of formats - see below. Any text beyond column 72 is discarded (in line with mainframe commenting standards) - the red bar indicates this limit.

TextPipe handles fields and groups with a fixed or variable number of repeats (e.g. OCCURS 8 TIMES. or OCCURS 6 INDEXED BY TN-IDX, OCCURS DEPENDING ON FIELDXXX).

Redefines are processed, but they are not output.

If you have an SAS copybook, then first use the filter mainframe\SAS to mainframe copybook.fll to convert it to the format above. If you file has field lengths then data type, use the filter mainframe\convert copybook format.fll to convert it to the correct format.

Output Format

You can choose if TextPipe outputs CSV (comma-separated) data, Tab delimited, Pipe-delimited, Fixed width or XML (filter only). If you use the wizard and need the final output in XML, use the Convert CSV to XML filter.

If TextPipe encounters an invalid packed or zoned decimal digit or sign field, it will output 'x' in the field to indicate an invalid value. If a packed or zoned field contains just spaces, TextPipe will output a blank field.

Quote All Alpha Values

When checked, all alpha values are quoted. When unchecked, only alpha values with embedded quotes or the delimiter are quoted

Assume adjacent COMP fields are overlapped

Odd-length COMP fields have half-bytes (nybbles) of unused data. Some compilers overlap adjacent COMP fields in files to save space. Check this option if your files contains overlapped COMP fields.

Your copybook must show the positions of every packed field and any extra filler COMP fields added by your compiler, as the location of these is system-dependant.

Trim leading spaces and zeroes from the output file

By default, TextPipe leaves leading spaces, zeros and even '+' signs on numerics. Check this option to remove them to save space in the output file.

Allow COMP-3 to be unsigned

Normally, COMP-3 fields have space allocated for a sign, regardless of whether it is used or not.

Check this option if you find that dates are coming out with 7 digits instead of 8. Often a copybook will specify dates as PIC S9(7) COMP-3 instead of PIC 9(8) COMP-3.

This option is supported by the wizard but not yet by the filter.

Only handle numeric fields

When checked, the wizard only generates filters to decode numeric fields.

Remove FILLER fields

When checked, the wizard removes data from fields with a name starting with the word 'FILLER'.

Add field name header

When checked, the wizard adds a file header to the output file.

Sample Formats (1)

TextPipe detects two main COBOL formats:

Both with and without line numbers:

03 TN-INSURANCE-DATA.

05 TN-INS-PRODUCTS OCCURS 6

07 TN-INS-PRODUCT-CODE PIC XX.

07 TN-INS-STATUS PIC X.

07 TN-INS-EFF-DATE PIC 9(8).

07 TN-INS-CANCELLATION-DATE PIC 9(8).

07 TN-INS-REINSTATEMENT-DATE PIC 9(8).

If you need to pass data through unchanged (ie binary data), then use the following construct:

03 COUNTY PIC X(3) usage is BINARY.

03 NUM PIC 9(3).

Note that the PIC field is 'X' ie Alpha. A PIC 9 field with BINARY implies a 2, 4 or 8 digit decimal.

If your code has extra line numbers or other data to the left of the levels (in Red below), please remove it. You can do this in TextPipe by using clicking the mouse at the top left, holding down Alt, and dragging a rectangular block to the bottom right, then press Delete:

05 DD-ACCT-TRSF-REL-IND

PICTURE X.

35766 05 FILLER PICTURE X.

47479 05 DD-ACCT-CMS-ESP PICTURE X.

05 DD-ACCT-BIRTH-DATE PICTURE S9(11) COMPUTATIONAL-3.

If TextPipe does not generate filters correctly, try these things:

1. Remove any extra information from the right hand margin
2. Ensure there is no word wrapping, and ensure every line ends in a period (.)
3. If dates come out with 7 digits instead of 8, check 'Allow COMP-3 to be unsigned'
4. If you have packed fields and the data alignment is not correct, try checking 'Assume adjacent COMP fields are overlapped'
5. If this fails, please send it to us for prompt attention.

Sample Formats (2)

This format without line numbers (the header is optional):

FIELD POSITION

DATA ELEMENT DEFINITION IN RECORD

RECORD CD 1 AN 1

PURCHASE REQUEST NUMBER 14 AN 2-15

PR LINE ITEM NUMBER 6 AN 16-21

QUANTITY 5 P 22-26

UNIT OF ISSUE 2 AN 27-28

STANDARD UNIT PRICE 5 P 29-33

FUND CLASS CODE 3 AN 34-36

STOCK NUMBER 15 AN 37-51


ASCII to EBCDIC

 ASCII to EBCDIC
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

Converts from ASCII to EBCDIC. EBCDIC is the character collating sequence (order) commonly used on mainframes. Some characters cannot be converted because they exist in one character set but not the other.

EBCDIC to ASCII and ASCII to EBCDIC conversion is usually done by the FTP program that performs the file transfer or download, but sometimes you may receive a file in the wrong format and there is no easy way to convert it. When the EBCDIC data file is moved down, ensure that it has correct blocking (mainframe talk for correct placement of CR\LF at end of every line). This is normally done when the file is moved down. Alternatively, use the Convert End Of Lines filter to put in line feeds manually.

If the ASCII to EBCDIC conversion doesn't suit you perfectly, use the Base Map On Filter List command to customize the translation.

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII

Expand Zoned Decimal

Expand Packed Decimal

Tutorials and User Guides (online)


EBCDIC to ASCII

 EBCDIC to ASCII
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

Converts from EBCDIC to ASCII. EBCDIC is the character collating sequence (order) commonly used on mainframes. Some characters cannot be converted because they exist in one character set but not the other.

If the EBCDIC to ASCII conversion doesn't suit you perfectly, use the Base Map On Filter List command to customize the translation.

If your document contains COMP fields or packed decimals, your output will contain garbage characters like curly braces {} at the end of numeric fields, and other special characters. Use the mainframe copybook wizard to specify the data structure of the input file.

See also our mainframe conversion guide online in the documents area.

See also:

ASCII to EBCDIC

Expand Zoned Decimal

Expand Packed Decimal

Tutorials and User Guides (online)


Expand Packed Decimal

 Expand Packed Decimal
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

This filter expands an EBCDIC packed decimal field (also known as Comp-3) to a raw EBCDIC number with a sign. Typically this filter is then followed by a Convert EBCDIC to ASCII filter - after all other fields have been expanded as well.

If your document contains COMP fields or packed decimals, use the mainframe copybook wizard.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

(*) Performance Note

If your numbers are always positive (i.e. no negatives) and there are no implied decimals, you will get better performance using the map Expand EBCDIC Packed Decimal.

All these filters can be found in the sample filter mainframe\generic mainframe file.fll

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII

ASCII to EBCDIC

Expand Zoned Decimal

Tutorials and User Guides (online)


Expand unsigned odd/even length decimal

 Expand unsigned odd/even length decimal
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

This filter expands an EBCDIC unsigned (as opposed to signed) packed decimal field to a raw EBCDIC number with no sign. Typically this filter is then followed by a Convert EBCDIC to ASCII filter - after all other fields have been expanded as well.

If your document contains COMP fields or packed decimals, use the mainframe copybook wizard, using the Treat 'COMP' as unsigned check box.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII

ASCII to EBCDIC

Expand Zoned Decimal

Tutorials and User Guides (online)


Expand Zoned Decimal

 Expand Zoned Decimal
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

This filter expands an EBCDIC zoned decimal field to a raw EBCDIC number with a sign. Typically this filter is then followed by a Convert EBCDIC to ASCII filter - after all other fields have been expanded as well.

If your document contains COMP fields or packed decimals, use the mainframe copybook wizard.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

(*) Performance Note

If your numbers are always positive (i.e. no negatives) and there are no implied decimals, you will get better performance using the map Expand EBCDIC Zoned Decimal.

All these filters can be found in the sample filter mainframe\generic mainframe file.fll

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII

ASCII to EBCDIC

Expand Packed Decimal

Tutorials and User Guides (online)


Expand Binary Number

 Expand Binary Number
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

This filter expands a series of digits stored in binary (BIG ENDIAN) form. The maximum width is 8 bytes.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.
3. If the data is stored in LITTLE ENDIAN order, use a Reverse filter inside the Restriction prior to the Expand Binary Numbers filter.

This page covers two filters - one outputs the result in ASCII, and the other outputs the result in EBCDIC (the assumption is that the data will later be converted to ASCII).

See also our mainframe conversion guide online in the documents area.

See also:

EBCDIC to ASCII

ASCII to EBCDIC

Expand Packed Decimal

Tutorials and User Guides (online)


Compress to Packed Decimal

 Compress to Packed Decimal
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

This filter compresses EBCDIC numeric data (optional leading sign, numbers and periods) to an EBCDIC packed decimal field (also known as Comp-3).

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
2. Compressing a field will decrease your output record length, so ensure you allow for this. A good strategy to avoid problems is to first compress the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

This filter will add hex 'B' to negative fields, hex 'C' to positive fields and hex 'F' to unsigned fields. If these codes don't match what your target needs, use a column or CSV restriction to apply a search/replace.

See also our mainframe conversion guide online in the documents area.

See also:

Compress to Zoned Decimal

EBCDIC to ASCII

ASCII to EBCDIC

Expand Zoned Decimal

Tutorials and User Guides (online)


Compress to Zoned Decimal

 Compress to Zoned Decimal
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Mainframe (Pro version only)

This filter compresses EBCDIC numeric data (optional leading sign, numbers and periods) to an EBCDIC zoned decimal field.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.
2. Compressing a field will decrease your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

See also our mainframe conversion guide online in the documents area.

See also:

Compress to Packed Decimal

EBCDIC to ASCII

ASCII to EBCDIC

Expand Packed Decimal

Tutorials and User Guides (online)


Unicode Filters

 Unicode Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

These filters perform a variety of Unicode functions.

Convert to and from Unicode (Standard and Pro)

ANSI to Unicode (Standard and Pro)

Unicode to ANSI (Standard and Pro)

Remove Unicode BOM (Byte Order Mark) (Standard and Pro)

Swap UTF-16 word order (Standard and Pro)

Swap UTF-32 word order (Standard and Pro)

Make Big Endian (Standard and Pro)

Make Little Endian (Standard and Pro)

Unicode Normalization functions:

NFC - Canonical Decomposition, followed by Canonical Composition (Standard and Pro)

NFD - Canonical Decomposition (Standard and Pro)

NFKD - Compatibility Decomposition (Standard and Pro)

NFKC - Compatibility Decomposition, followed by Canonical Composition (Standard and Pro)

Compose (Standard and Pro)

These filters were previously found in the Convert Menu. See also the Unicode Restriction filters.


Unicode and Code Page conversion

 Unicode and Code Page conversion
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

This filter converts between a variety of text encodings, including various forms of Unicode, and a variety of code pages.

Note: The Trial Output Area can only display ANSI and Unicode text. It cannot understand or display other code pages. If you want to check an output file, you MUST use a program that is specifically enabled for the code page you want to view. e.g. if you convert Unicode to CP936, the Trial Output will show rubbish. You must use a CP936-enabled application to view the output file.

Convert FROM

Enter the source or input encoding here.

Both the Convert FROM and Convert TO lists contain:

1. Inbuilt Conversions - provided by TextPipe's internal libraries
2. Windows Installed Code Pages - provided by Windows. The code pages available depend on which code pages you have installed (see Control Panel\Regional Settings).

In some cases the conversions offered overlap. In general, the Inbuilt conversions will be faster than the Windows conversions, but in some cases they may not be as up-to-date, particularly for South-East Asian languages.

Convert TO

Enter the destination or output encoding here.

If a character cannot be converted to the destination format, the default character for the encoding (or a space) is output instead. This maintains the column spacing for reports.

If the target Unicode format specifies a byte order (e.g. UTF-16LE, UTF016BE), then no Byte Order Mark will be output. If you require a Byte Order Mark, follow this filter with an Add Header filter.

Swap

This swaps the Convert FROM and Convert TO settings.

Error String

This character or string is output when there is no suitable character in the destination encoding. It defaults to a space.

Unicode Encodings

To find out the encoding of a file, drag it to TextPipe's File Grid, right-click the file and choose Analyze File.

For most purposes, Unicode has three main encodings:

· UTF-8 - most common for web pages
· UTF-16LE (little endian) - most common on Windows
· UTF-16BE (big endian) - less common on Windows.

Note: Unicode itself can be stored in a variety of formats, including UTF-* and UCS-*, with the most common being UTF-16 and UTF-8.

Trial Run Area

For Unicode Input from the Trial Run area, set Convert FROM to 'UTF16-LE', and on the Trial Run Area tab, check Treat Trial Run Input as Unicode (UTF16-LE) instead of ANSI.

For Unicode Output to the Trial Run area, set Convert TO to 'UTF16-LE', and on the Trial Run Area tab, check Treat Trial Run Output as Unicode (UTF16-LE) instead of ANSI.

If your input or output is UTF-8, ensure the checkbox is unchecked.

Conversions offered

See also: Code page conversions offered

ARMSCII-8

ASCII

ANSI

BIG5

BIG5HKSCS

C99

CES-BIG5

CES-GBK

CNS-11643-1

CNS-11643-15

CNS-11643-2

CNS-11643-3

CNS-11643-4

CNS-11643-5

CNS-11643-6

CNS-11643-7

CNS-11643-INV

CP1046

CP1124

CP1125

CP1129

CP1133

CP1161

CP1162

CP1163

CP1250

CP1251

CP1252

CP1253

CP1254

CP1255

CP1256

CP1257

CP1258

CP437

CP737

CP775

CP850

CP852

CP853

CP855

CP856

CP857

CP858

CP860

CP861

CP862

CP863

CP864

CP865

CP866

CP869

CP874

CP922

CP932

CP949

CP950

DEC-Hanyu

DEC-Kanji

EUC-CN

EUC-jisx0213

EUC-JP

EUC-KR

EUC-TW

GB18030

GB2312

GBK

GEORGIAN-ACADEMY

GEORGIAN-PS

HP-ROMAN8

HZ

ISO-2022-CN

ISO-2022-CN-EXT

ISO-2022-JP

ISO-2022-JP1

ISO-2022-JP2

ISO-2022-JP3

ISO-2022-KR

ISO646-CN

ISO-646-JP

ISO-8859-1

ISO-8859-10

ISO-8859-13

ISO-8859-14

ISO-8859-15

ISO-8859-16

ISO-8859-2

ISO-8859-3

ISO-8859-4

ISO-8859-5

ISO-8859-6

ISO-8859-7

ISO-8859-8

ISO-8859-9

ISO-IR-165

JAVA

JIS_X0201

JIS_X0208

JIS_X0212

JOHAB

Johab-Hangul

KOI8-R

KOI8-RU

KOI8-T

KOI8-U

KSC-5601

MacArabic

MacCentralEurope

MacCroatian

MacCyrillic

MacGreek

MacHebrew

MacIceland

MacRoman

MacRomania

MacThai

MacTurkish

MacUkraine

MULELAO-1

NEXTSTEP

RISCOS1

SHIFT-JISX0213

SJIS

TCVN

TDS565

TIS-620

UCS-2

UCS-2BE

UCS-2-Internal

UCS-2LE

UCS-2-Swapped

UCS-4

UCS-4BE

UCS-4-Internal

UCS-4LE

UCS-4-Swapped

UTF-16

UTF-16BE

UTF-16LE

UTF-32

UTF-32BE

UTF-32LE

UTF-7

UTF-8

VISCII

Note:

· UCS-4 is UTF-32 with support for code points beyond U+10FFFF (which are supposed to be unassignable forever).
· UCS-2 is UTF-16 with surrogate support removed (so code points beyond U+FFFF cannot be represented).

Equivalent encodings

The following list gives equivalent encodings (different names for the same encoding), one per line.

US-ASCII,ASCII,ISO646-US,ISO_646.IRV:1991,ISO-IR-6,ANSI_X3.4-1968,ANSI_X3.4-1986,CP367,IBM367,US,csASCII

UCS-2,ISO-10646-UCS-2,csUnicode

UCS-2BE,UNICODEBIG,UNICODE-1-1,csUnicode11

UCS-2LE,UNICODELITTLE

UCS-4,ISO-10646-UCS-4,csUCS4

UTF-7,UNICODE-1-1-UTF-7,csUnicode11UTF7

ISO-8859-1,ISO_8859-1,ISO_8859-1:1987,ISO-IR-100,CP819,IBM819,LATIN1,L1,csISOLatin1,ISO8859-1

ISO-8859-2,ISO_8859-2,ISO_8859-2:1987,ISO-IR-101,LATIN2,L2,csISOLatin2,ISO8859-2

ISO-8859-3,ISO_8859-3,ISO_8859-3:1988,ISO-IR-109,LATIN3,L3,csISOLatin3,ISO8859-3

ISO-8859-4,ISO_8859-4,ISO_8859-4:1988,ISO-IR-110,LATIN4,L4,csISOLatin4,ISO8859-4

ISO-8859-5,ISO_8859-5,ISO_8859-5:1988,ISO-IR-144,CYRILLIC,csISOLatinCyrillic,ISO8859-5

ISO-8859-6,ISO_8859-6,ISO_8859-6:1987,ISO-IR-127,ECMA-114,ASMO-708,ARABIC,csISOLatinArabic,ISO8859-6

ISO-8859-7,ISO_8859-7,ISO_8859-7:1987,ISO-IR-126,ECMA-118,ELOT_928,GREEK8,GREEK,csISOLatinGreek,ISO8859-7

ISO-8859-8,ISO_8859-8,ISO_8859-8:1988,ISO-IR-138,HEBREW,csISOLatinHebrew,ISO8859-8

ISO-8859-9,ISO_8859-9,ISO_8859-9:1989,ISO-IR-148,LATIN5,L5,csISOLatin5,ISO8859-9

ISO-8859-10,ISO_8859-10,ISO_8859-10:1992,ISO-IR-157,LATIN6,L6,csISOLatin6,ISO8859-10

ISO-8859-13,ISO_8859-13,ISO-IR-179,LATIN7,L7,ISO8859-13

ISO-8859-14,ISO_8859-14,ISO_8859-14:1998,ISO-IR-199,LATIN8,L8,ISO-CELTIC,ISO8859-14

ISO-8859-15,ISO_8859-15,ISO_8859-15:1998,ISO-IR-203,ISO8859-15

ISO-8859-16,ISO_8859-16,ISO_8859-16:2000,ISO-IR-226,ISO8859-16

KOI8-R,csKOI8R

CP1250,WINDOWS-1250,MS-EE

CP1251,WINDOWS-1251,MS-CYRL

CP1252,WINDOWS-1252,MS-ANSI

CP1253,WINDOWS-1253,MS-GREEK

CP1254,WINDOWS-1254,MS-TURK

CP1255,WINDOWS-1255,MS-HEBR

CP1256,WINDOWS-1256,MS-ARAB

CP1257,WINDOWS-1257,WINBALTRIM

CP1258,WINDOWS-1258

CP850,IBM850,850,csPC850Multilingual

CP862,IBM862,862,csPC862LatinHebrew

CP866,IBM866,866,csIBM866

MACINTOSH,MAC,csMacintosh

HP-ROMAN8,ROMAN8,R8,csHPRoman8

CP1133,IBM-CP1133

TIS-620,TIS620,TIS620-0,TIS620.2529-1,TIS620.2533-0,TIS620.2533-1,ISO-IR-166

CP874,WINDOWS-874

VISCII,VISCII1.1-1,csVISCII

TCVN,TCVN-5712,TCVN5712-1,TCVN5712-1:1993

JIS_C6220-1969-RO,ISO646-JP,ISO-IR-14,JP,csISO14JISC6220ro

JIS_X0201,JISX0201-1976,X0201,csHalfWidthKatakana

JIS_X0208,JIS_X0208-1983,JIS_X0208-1990,JIS0208,X0208,ISO-IR-87,JIS_C6226-1983,csISO87JISX0208

JIS_X0212,JIS_X0212.1990-0,JIS_X0212-1990,X0212,ISO-IR-159,csISO159JISX02121990

GB_1988-80,ISO646-CN,ISO-IR-57,CN,csISO57GB1988

GB_2312-80,ISO-IR-58,csISO58GB231280,CHINESE

ISO-IR-165,CN-GB-ISOIR165

KSC_5601,KS_C_5601-1987,KS_C_5601-1989,ISO-IR-149,csKSC56011987,KOREAN

EUC-JP,EUCJP,Extended_UNIX_Code_Packed_Format_for_Japanese,csEUCPkdFmtJapanese

SHIFT_JIS,SHIFT-JIS,SJIS,MS_KANJI,csShiftJIS

ISO-2022-JP,csISO2022JP

ISO-2022-JP-2,csISO2022JP2

EUC-CN,EUCCN,GB2312,CN-GB,csGB2312

GBK,CP936

ISO-2022-CN,csISO2022CN

HZ,HZ-GB-2312

EUC-TW,EUCTW,csEUCTW

BIG5,BIG-5,BIG-FIVE,BIGFIVE,CN-BIG5,csBig5

BIG5-HKSCS,BIG5HKSCS

EUC-KR,EUCKR,csEUCKR

CP949,UHC

JOHAB,CP1361

ISO-2022-KR,csISO2022KR

See also

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion

Code pages

Some of these conversions are provided by the libiconv library. The remainder are provided by Windows.


ANSI to Unicode

 ANSI to Unicode
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Converts from ANSI to Unicode. Converts single byte ANSI characters to double byte Unicode characters. This filter can be useful if you want to send a text file to someone using a language other than your own.

This function was originally included in older versions of TextPipe. To use this feature in conjunction with the new Unicode conversion functions, you must first convert from UTF-16LE to Unicode.

On our test system, US ASCII is transformed to UTF-16LE:

00000000 20 00 09 00 20 00 33 00 32 00 0D 00 0A 00 21 00 ... .3.2.....!.

00000010 09 00 20 00 33 00 33 00 0D 00 0A 00 22 00 09 00 .. .3.3....."...

00000020 20 00 33 00 34 00 0D 00 0A 00 23 00 09 00 20 00 .3.4.....#... .

00000030 33 00 35 00 0D 00 0A 00 24 00 09 00 20 00 33 00 3.5.....$... .3.

This filter is often followed by an Add Header filter, to add a Unicode byte order mark (BOM), \xFF\xFE.

See also

Unicode to ANSI

Unicode conversion


Unicode to ANSI

 Unicode to ANSI
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Converts from Unicode to ANSI. Converts double byte Unicode characters to single byte ANSI characters. This filter can be useful if you want to send a text file to someone using a language other than your own.

This function was originally included in older versions of TextPipe. To use this feature in conjunction with the new Unicode conversion functions, you must first convert from UTF-16LE to Unicode.

This filter is often followed by a Remove start or end of file filter, to either remove the first two bytes of Unicode (before the conversion) or the first byte of ANSI (after the conversion), to remove the leading Unicode byte order mark (BOM).

See also

ANSI to Unicode


UTF16-LE to Escaped Unicode (ASCII + UCN \U)

 UTF16-LE to Escaped Unicode (ASCII + UCN \U)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

This filter converts incoming Unicode text in UTF16-LE format to ASCII output with embedded \uXXXX hex control codes. If the input is not UTF16-LE, first convert it with Unicode format conversion filter.

in the Filter list.

??????????

to:

in the Filter list.

\u0647\u062E\u062D\u062C\u0644\u4E14\u4E55\u4E5C\u4EB6\u4EB6

To convert back to UTf-8, use the example filter etext\Escaped Unicode (ASCII + UCN) to UTF-8.fll


Swap UTF-32 word order

 Swap UTF-32 word order
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro). This filter swaps groups of 2-byte words e.g.

Byte number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
Input File | FF | FE | 00 | 00 | 00 | 31 | 00 | 00
Output File | 00 | 00 | FE | FF | 00 | 00 | 31 | 00

This is commonly used to transform big-endian or little-endian Unicode files so that other programs can use them.

See also

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion


Swap UTF-16 word order

 Swap UTF-16 word order
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro). This filter swaps pairs of bytes e.g.

Byte number | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8
Input File | FF | FE | 00 | 20 | 00 | 31 | 00 | 32
Output File | FE | FF | 20 | 00 | 31 | 00 | 32 | 00

This is commonly used to transform big-endian or little-endian Unicode files so that other programs can use them.

See also

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion


Remove BOM

 Remove BOM
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

This filter removes the Unicode Byte Order Mark from the start of Unicode files, if it is present.

Bytes removed | Description
00 00 FE FF | UTF-32, big-endian
FF FE 00 00 | UTF-32, little-endian
FE FF | UTF-16, big-endian
FF FE | UTF-16, little-endian
EF BB BF | UTF-8

See also

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion

Remove menu

Remove columns

Remove delimited fields (CSV, Tab, Pipe etc)


Make Big Endian

 Make Big Endian
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro). Makes a Little Endian Unicode file into a Big Endian Unicode file e.g.

Input file | Output file
00 00 FE FF 00 00 00 4D | Unchanged
FE FF 4E 8C | Unchanged
FF FE 00 00 4D 00 00 00 | 00 00 FE FF 00 00 00 4D
FF FE 8C 4E | FE FF 4E 8C

Note - the file MUST start with a Byte Order Mark for it to be correctly identified.

See also

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion


Make Little Endian

 Make Little Endian
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro). Makes a Big Endian Unicode file into a Little Endian Unicode file e.g.

Input file | Output file
00 00 FE FF 00 00 00 4D | FF FF 00 00 4D 00 00 00
FE FF 4E 8C | FF FE 8C 4E
FF FE 00 00 4D 00 00 00 | Unchanged
FF FE 8C 4E | Unchanged

Note - the file MUST start with a Byte Order Mark for it to be correctly identified.

See also

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion


Normalize NFC

 Normalize NFC
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Applies a Unicode NFC - Canonical Decomposition, followed by Canonical Composition transformation to incoming Unicode text (UTF16-LE).

Output is also Unicode UTF16-LE.

See also:

NFC - Canonical Decomposition, followed by Canonical Composition

NFD - Canonical Decomposition

NFKD - Compatibility Decomposition

NFKC - Compatibility Decomposition, followed by Canonical Composition

Compose


Normalize NFD

 Normalize NFD
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Applies a Unicode NFD - Canonical Decomposition transformation to incoming Unicode text (UTF16-LE).

Output is also Unicode UTF16-LE.

See also:

NFC - Canonical Decomposition, followed by Canonical Composition

NFD - Canonical Decomposition

NFKD - Compatibility Decomposition

NFKC - Compatibility Decomposition, followed by Canonical Composition

Compose


Normalize NFKC

 Normalize NFKC
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Applies a Unicode NFKC - Compatibility Decomposition, followed by Canonical Composition transformation to incoming Unicode text (UTF16-LE).

Output is also Unicode UTF16-LE.

See also:

NFC - Canonical Decomposition, followed by Canonical Composition

NFD - Canonical Decomposition

NFKD - Compatibility Decomposition

NFKC - Compatibility Decomposition, followed by Canonical Composition

Compose


Normalize NFKD

 Normalize NFKD
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Applies a Unicode NFKD - Compatibility Decomposition transformation to incoming Unicode text (UTF16-LE).

Output is also Unicode UTF16-LE.

See also:

NFC - Canonical Decomposition, followed by Canonical Composition

NFD - Canonical Decomposition

NFKD - Compatibility Decomposition

NFKC - Compatibility Decomposition, followed by Canonical Composition

Compose


Compose

 Compose
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Unicode (Standard and Pro)

Applies a Unicode Compose transformation to incoming Unicode text (UTF16-LE).

Output is also Unicode UTF16-LE.

See also:

NFC - Canonical Decomposition, followed by Canonical Composition

NFD - Canonical Decomposition

NFKD - Compatibility Decomposition

NFKC - Compatibility Decomposition, followed by Canonical Composition

Compose


Add Filters

 Add Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add

These filters add text or reformat it.

Add line numbers

Add left margin

Add right margin

Add header and Add file header

Add footer and Add file footer

Add column of text

Add bytes

Add text side by side

Add repeating text side by side

Word wrap

Pad to width (Pro version only)

Center in width

Right justify in width

Truncate to width (Pro version only)

Force to width (Pro version only)

To add new filters to the filter list, select one from the Filter menus.


Add line numbers

 Add line numbers
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add

This filter adds line numbers to the start of every line. By default, each line number is five digits (with leading zeroes inserted where necessary), followed by a space, but this can be tailored.

The starting line number and the line number increment can both be entered. The maximum value for each field is 2147483647.

Format

The format field allows the output of the number to be altered, which can be used for including static text on either side of the number or altering the appearance of the number (e.g. leading zeros, hexadecimal output etc). For more information on the allowed options, please see Format Specifiers. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters.

Don't Number Blank Lines

When this field is checked, blank lines do not have a line number printed on the left hand side. This can be used to make the output look cleaner and break up sections of code.

Skip Increment on Blank Lines

When this field is checked, blank lines are not counted.

Don't Number Blank Lines must be checked for this field to be enabled.

Don't Reset at end of file

When this field is checked, the line number does not get reset to First line number when the end of a file is reached. Instead, it keeps counting. This can be very useful when numbering inside a restriction or sub filter, as these both treat the restricted text as a new file.


Left margin

 Left margin
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add

Adds the contents of the file or the specified text to the start of every line. Useful for adding email reply quotes eg. '> ' to every line. It can also be used to indent text. Sub filters can be used to pre-process the file or text being added.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Note: An empty file or restriction has no lines, so no left margin will be added. Use an Add Footer filter instead. The same applies after a Remove All filter.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to right margin

Add header text

Add footer text

Add column of text


Right margin

 Right margin
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add

Adds the contents of the file or the specified text to the end of every line. You can use this filter after a Pad To Width filter to put characters in the right hand margin. Sub filters can be used to pre-process the file or text being added.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Note: An empty file or restriction has no lines, so no right margin will be added. Use an Add Header filter instead. The same applies after a Remove All filter.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add header text

Add footer text

Add column of text


Header

 Header
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Note: This filter outputs text both at the start of a file, and the start of a Restriction. To only output text at the start of a file, use the Add File Header filter.

Adds the contents of the file or the specified text to the start of each file. Useful for adding advertising banners to web pages or other forms of file header. This filter is usually positioned at the top of the filter list ie just after <input>. Sub filters can be used to pre-process the file or text being added.

Note: If you wish to add a header to each page of a file that contains many pages, the best method of doing so is to use an 'Add Header Text' filter in conjunction with a search and replace for page end characters. Search for a form feed character (\f or \012), and when you find it, replace with a \f followed by your page header text.

You can use this filter to add a Unicode Byte Order Mark (BOM) to your file:

Byte sequence | Description
\x00\x00\xFE\xFF | UTF-32, big-endian
\xFF\xFE\x00\x00 | UTF-32, little-endian
\xFE\xFF | UTF-16, big-endian
\xFF\xFE | UTF-16, little-endian
\xEF\xBB\xBF | UTF-8 - optional, as UTF-8 only has one byte order. Mainly used as a Unicode signature.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header Add file header

Add footer Add file footer

Add column of text


Add File Header

 Add File Header
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Note: Although otherwise identical to the Add Header filter, this filter does NOT generate any output at the start of a Restriction.

Adds the contents of the file or the specified text to the start of each file. Useful for adding advertising banners to web pages or other forms of file header. This filter is usually positioned at the top of the filter list ie just after <input>. Sub filters can be used to pre-process the file or text being added.

Note: If you wish to add a header to each page of a file that contains many pages, the best method of doing so is to use an 'Add Header Text' filter in conjunction with a search and replace for page end characters. Search for a form feed character (\f or \012), and when you find it, replace with a \f followed by your page header text.

You can use this filter to add a Unicode Byte Order Mark (BOM) to your file:

Byte sequence | Description
\x00\x00\xFE\xFF | UTF-32, big-endian
\xFF\xFE\x00\x00 | UTF-32, little-endian
\xFE\xFF | UTF-16, big-endian
\xFF\xFE | UTF-16, little-endian
\xEF\xBB\xBF | UTF-8 - optional, as UTF-8 only has one byte order. Mainly used as a Unicode signature.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header Add file header

Add footer Add file footer

Add column of text


Footer

 Footer
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Note: This filter outputs text both at the end of a file, and the end of a Restriction. To only output text at the end of a file, use the Add File Footer filter.

Adds the contents of the file or the specified text to the end of each file. This filter is usually positioned at the bottom of the filter list ie just before <output>. Sub filters can be used to pre-process the file or text being added.

Note: If you wish to add a footer to the end of each page of a file that contains many pages, the best method of doing so is to use an 'Add Footer Text' filter in conjunction with a search and replace for page end characters. Search for a form feed character (\f or \012), and when you find it, replace with your page footer text followed by \f.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Useful for adding copyright messages to the end of HTML files.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header Add file header

Add footer Add file footer

Add column of text


Add file footer

 Add file footer
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Although otherwise identical to the Add Footer filter, this filter does NOT generate any output at the end of a Restriction.

Adds the contents of the file or the specified text to the end of each file. This filter is usually positioned at the bottom of the filter list ie just before <output>. Sub filters can be used to pre-process the file or text being added.

Note: If you wish to add a footer to the end of each page of a file that contains many pages, the best method of doing so is to use an 'Add Footer Text' filter in conjunction with a search and replace for page end characters. Search for a form feed character (\f or \012), and when you find it, replace with your page footer text followed by \f.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables. Special characters and macros are ONLY supported when they are entered in the text field - they will not be interpreted or parsed when a file's contents are included.

Useful for adding copyright messages to the end of HTML files.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header Add file header

Add footer Add file footer

Add column of text


Add column of text

 Add column of text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

Inserts a new column of text. The position the text is inserted is determined by a column count. The leftmost column is column 1 – inserting in this column displaces all other text to the right. If the insert column given is 0, the text is inserted at the end of the line. If the insert column is negative, the text is inserted at the given position relative to the end of the line. If the insert column given is before the start of the line, or beyond the end of the line, then the text is pre-pended or appended to the line respectively. Sub filters can be used to pre-process the file or text being inserted.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header text

Add footer text


Add line of text
 Add line of text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Inserts a new line of text. The position the line is inserted is determined by a line count, with the first line being 1. Sub filters can be used to pre-process the file or text being inserted.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header text

Add footer text


Add bytes

 Add bytes
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

Inserts bytes at a given offset, from 0 up to the size of the file. Sub filters can be used to pre-process the file or text being inserted.

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Add text to left margin

Add text to right margin

Add header text

Add footer text


Add text side by side

 Add text side by side
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter is used to join two files together as columns side by side. If File A represents each file from the Files to Process list, and File B is the file being added side by side, then each line of File B gets added to the right hand end of each line from File A. If File A has more lines than File B, then the lines at the end of File A won't have anything from File B (if you need them to, either make sure that both files have the same number of lines, or use the Add Repeating Text Side By Side filter). If File B has more lines than File A, then the extra lines from File B will be omitted. To avoid this, pad File A with extra lines.

You may wish to precede the Add Text Side By Side filter with a Pad to Width filter to make sure that the right hand margin of File A is not ragged when the lines are joined.

Sub filters can be used to pre-process the file or text being added.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See Also

Add Repeating Text Side By Side

Pad to Width


Add repeating text side by side

 Add repeating text side by side
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter is used to join two files together as columns side by side. If File A represents each file from the Files to Process list, and File B is the file being added side by side, then each line of File B gets added to the right hand end of each line from File A. If File A has more lines than File B, then File B is repeated. If File B has more lines than File A, then the extra lines from File B will be omitted.

You may wish to precede the Add Repeating Text Side By Side filter with a Pad to Width filter to make sure that the right hand margin of File A is not ragged when the lines are joined.

Sub filters can be used to pre-process the file or text being added.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See Also

Add Text Side By Side

Pad to Width


Word wrap

 Word wrap
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter reformats text that is wider than the specified column width. Use it when text has been formatted without line feeds.

The line feed added by this filter is a DOS CR/LF pair, so if you need a Unix or Mac end of line character you will need to follow it with a Convert End of Lines filter.

If you need to add line feeds to a document with fixed record sizes, use a Convert End of Lines filter.


Pad to width

 Pad to width
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter ensures that all lines are of a uniform length by adding spaces to lines that are shorter than the given width.

See also

Center in width

Right justify in width


Truncate to width

 Truncate to width
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter truncates lines that are longer than the specified width.

See also

Center in width

Right justify in width


Center in width

 Center in width
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter centres text in a line of the given width by adding spaces at the start and end of the line. Lines longer than this width are ignored.

See also

Pad to width

Right justify in width


Right justify in width

 Right justify in width
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter right justifies text in a line of the given width by adding spaces at the start of the line. Lines longer than this width are ignored.

See also

Pad to width

Center in width


Force to width

 Force to width
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Add (Std and Pro version only)

This filter truncates lines that are longer than the specified width.

See also

Center in width

Right justify in width


Remove Filters

 Remove Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove

These filters remove unwanted information.

Remove Blanks From Start Of Line

Remove Blanks From End Of Line

Remove blank lines

Remove multiple whitespace

Remove HTML and XML\Remove all tags

Remove HTML and XML\Remove elements (e.g. <table...>

Remove HTML and XML\Remove attributes (e.g. height="...")

Remove HTML and XML\Remove tag pair (e.g. <script>...</script>)

Remove backspaces

Resolve backspaces

Remove line range

Remove lines matching

Remove lines not matching

Remove lines from list (Pro version only)

Remove lines matching pattern list (Pro version only)

Remove duplicate lines

Remove start or end of file (Pro version only)

Remove columns (Pro version only)

Remove delimited fields (CSV, Tab, Pipe etc) (Pro version only)

Remove byte range (Pro version only)

Remove binary characters (Pro version only)

Remove ANSI codes (Pro version only)

Remove Unicode BOM (Byte Order Mark)

Remove email headers

Remove all


Remove Blanks From Start Of Line

 Remove Blanks From Start Of Line
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove

Removes spaces and tabs from the start of every line. This is equivalant to an unindent action.

See also

Remove Blanks From End Of Line

Remove blank lines

Remove multiple whitespace

Remove menu


Remove Blanks From End Of Line

 Remove Blanks From End Of Line
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove

Removes spaces and tabs from the end of every line. In general, these characters are never used and can be removed with no ill-effects.

See also

Remove Blanks From Start Of Line

Remove blank lines

Remove multiple whitespace

Remove menu


Remove blank lines

 Remove blank lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove

Removes blank lines. Note, lines with spaces or tabs are not removed. Use the Remove Blanks From Start Of Line filters first to rectify this.

See also

Remove Blanks From Start Of Line

Remove Blanks From End Of Line

Remove multiple whitespace

Remove menu


Remove multiple whitespace

 Remove multiple whitespace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove

Removes sequences of multiple spaces or tabs and replaces them with a single space. Useful for making documents look professional – multiple spaces look untidy.

See also

Remove Blanks From Start Of Line

Remove Blanks From End Of Line

Remove blank lines

Remove menu


Remove HTML and SGML

 Remove HTML and SGML
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

Use this filter to convert HTML documents to a readable format.

This filter removes HTML and XML markup tags i.e. everything including and between <> brackets.

If your file still has HTML special characters in it such as

&nbsp;

&#174;

etc, follow this filter with the included filter "html\HTML to text.fll" filter. This filter will replace the escaped special characters e.g. &#174; with (tm).

See also

Remove Blanks From Start Of Line

Remove Blanks From End Of Line

Remove blank lines

Remove menu


Remove XML field

 Remove XML field
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter is used to remove HTML/XML-delimited elements, attributes, and text between tag pairs. A Remove All sub filter is automatically added.

The filter fields are identical to a HTML/XML restriction filter.

For example, if your data looked like this, and you wanted to remove the CompanyName field:

<TextPipeRow>

<CustomerID>ALFKI</CustomerID>

<CompanyName>Alfreds Futterkiste</CompanyName>

<ContactName>Maria Anders</ContactName>

<ContactTitle>Sales Representative</ContactTitle>

<Address>Obere Str. 57</Address>

<City>Berlin</City>

<Region></Region>

<PostalCode>12209</PostalCode>

<Country>Germany</Country>

<Phone>030-0074321</Phone>

<Fax>030-0076545</Fax>

</TextPipeRow>

Specify the 'Restrict to tag' name to remove as 'CompanyName'

See also

Remove menu

Remove columns

Remove delimited fields (CSV, Tab, Pipe etc)


Remove backspaces

 Remove backspaces
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

Remove backspaces ie all ASCII code 8s.

See also

Resolve backspaces

Remove menu


Resolve backspaces

 Resolve backspaces
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

Resolve backspaces ie remove both the backspaces and the characters prior to the backspaces that would have been deleted.

See also

Remove backspaces

Remove menu


Remove matching/non-matching lines

 Remove matching/non-matching lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

TextPipe will remove any lines that match/don't match the given pattern. For more options, please see the Restrict to matching lines filter. Unlike that filter, this filter is not allowed to have any sub filters.

Note - this filter will not match blank lines.

See also

Retain matching lines (grep)/non-matching lines (inverse grep)

Retain lines found in file

Retain lines matching pattern list

Remove menu

Perl pattern matching reference

EasyPattern reference


Remove lines from list

 Remove lines from list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

TextPipe will remove all lines that EXACTLY match any of the lines in the given text file, or in the text memo area (line breaks can be of any format).

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Remove lines matching pattern list

Retain lines found in file

Remove lines matching

Remove menu


Remove lines

 Remove lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter removes a range of lines.

Several different means of specifying the lines to remove are provided:

1. Start line, End line. This removes all lines including and between the specified lines.
2. Start line, Length. Removes Length lines starting from (and including) line Start.
3. End line (relative), Length. Removes Length lines backwards starting from (and including) line End.
4. Start line to End line (relative). Removes all lines from line Start to line End lines from the end of file.
5. Start line (relative) to End line (relative). Removes Start lines from the end of file to End lines from end of file.

See also

Retain lines found in file

Retain lines matching pattern list

Remove menu


Remove lines matching pattern list

 Remove lines matching pattern list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

TextPipe will remove all lines that match any of the patterns in the given text file (or specified in the text memo area). Each pattern must be specified on a line of its own (line breaks can be of any format). Each line is interpreted as a perl-style pattern, with options of case-insensitive matching, non-greedy matching, comments not allowed. If you need to change these options, precede each line by the following codes:

Change option command | Result
(?-i) | Case-sensitive matching
(?x) | Comments allowed
(?U) | Greedy matching

Note - this filter will not match blank lines.

See also

Retain lines found in list

Retain matching lines from list

Remove menu


Remove duplicate lines

 Remove duplicate lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

Removes duplicate lines - even if they are not next to each other. From a set of duplicate lines, the line that gets output is the first duplicate line of the set.

Ignore case

If ignore case is checked, lines do not need to be cased identically to be considered duplicates. Two identical lines, one in upper case, and one in lower case, would be considered duplicates and removed by this filter. If ignore case is unchecked, the lines must be identical to be considered duplicates. The case checking routines are ANSI aware, so their behaviour may change depending on your locale.

Start column

The Start Column field allows you to ignore leading characters before a comparison is made. This can be very useful for ignoring line numbers is source code etc.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

See also

Count duplicate lines

Remove menu


Remove start or end of file (head/tail)

 Remove start or end of file (head/tail)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter is used to include or exclude either lines or bytes from the start or end of a file. It performs an extension of the functionality of the Unix head and tail commands.

Include or Exclude

Specifies whether to keep or remove the selected text. Normally you know how much text you wish to include or exclude eg 10 lines.

Counting Units

Specifies the quantity of text and how it is to be counted, in lines or bytes.

Where to Count From

Start Of File and From End Of File - specify where the text should be found – at the start or end of the processed text.

See also

Remove menu


Remove columns

 Remove columns
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter is used to remove columns of text, given a column specification that describes the position of the column relative to the start or end of the line, and the width of the column.

Several different means of specifying the columns to remove are provided:

1. Start column, End column. This removes all text including and between the specified columns. Useful for removing column in fixed width data files.
2. Start column, Length. Removes Length characters starting from (and including) column Start.
3. End column, Length. Removes Length characters backwards starting from (and including) column End.
4. Start column to end of line. Removes all characters from the Start column to the very end of the line. Useful for making a file a uniform width.
5. Length to end of line. Removes Length characters backwards starting from (and including) the last column.

Note - if you are removing more than one column range, it is easiest to remove ranges from right-to-left so that the position of the columns doesn't change.

See also

Remove menu

Remove columns

Remove delimited fields (CSV, Tab, Pipe etc)


Remove delimited fields (CSV, Tab, Pipe etc)

 Remove delimited fields (CSV, Tab, Pipe etc)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter is used to remove fields delimited by a given character.

Choose the delimiter that separates your fields

You can choose a predefined delimiter character, or select your own. The trailing delimiter (if any) is also removed. When Comma (.csv) is chosen, TextPipe automatically handles single and double quoted strings, with embedded line feeds.

First Row Contains Field Names

If the first line of the file contains Field Names, check this box so that TextPipe can count how many fields are expected. It can then determine if a field has embedded CR/LF characters and spans multiple lines. TextPipe can also determine this without a header if the fields are properly double-quoted - TextPipe will notice the missing double quote and continue reading the record from the following line.

Remove Fields

Several different means of specifying the fields to remove are provided:

1. Start field, end field. This removes all text including and between the specified fields.
2. Start field, Length. Removes Length fields starting from (and including) field Start.
3. End field, Length. Removes Length fields backwards starting from (and including) field End.
4. Start field to end of line. Removes all fields from the Start field to the very end of the line.
5. Length to end of line. Removes Length fields backwards starting from (and including) the last field.

Note - if you are removing more than one field range, it is easiest to remove ranges from right-to-left so that the position of the fields doesn't change.

See also

Remove menu

Remove columns


Remove byte range

 Remove byte range
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter is used to remove a range of bytes.

Several different means of specifying the bytes to remove are provided:

1. Start byte, end byte. This removes all text including and between the specified bytes.
2. Start byte, Length. Removes Length bytes starting from (and including) the start byte.
3. End byte, Length. Removes Length bytes backwards starting from (and including) byte End.
4. Start byte to end of file. Removes all bytes from the Start byte to the very end of the file.
5. Length to end of file. Removes Length bytes backwards starting from (and including) the last byte.

Note - if you are removing more than one byte range, it is easiest to remove ranges from right-to-left so that the position of the bytes doesn't change.

See also

Remove menu

Remove columns

Remove CSV and Tab fields


Remove binary characters

 Remove binary characters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

Removes binary characters such as those higher than ASCII code 127, and those less than ASCII code 32 except for carriage returns (ASCII code 13) and line feeds (ASCII code 10).

This filter if very useful if you have a corrupted word processor file, or if you just want to see what text, if any, is inside a file. The binary information required by the word processor can be removed, leaving you with just the text.

If you wish to retain foreign characters, create a new map based on the Remove Binary filter, then customize the map to retain the characters you need.

See also

Remove menu


Remove ANSI codes

 Remove ANSI codes
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

ANSI (American National Standards Institute) codes are included in various streams of information (most commonly BBS connections), to provide a remote computer with control over cursor positioning, text attributes, etc. They are also used in connections between minicomputers and mainframe computers and the terminals connected to them.

The need to use an ANSI filter can be recognized when something like the following example shows up in a file viewed in a text editor:

<[0;1;4mas<[m - MC88000 assembler

In this example the "as" near the beginning is displayed in a different color than the rest of the line when the ANSI codes are properly processed. The Escape (ASCII 27) codes above have been replaced by the < symbol to make this line printable.

The Remove ANSI Escape Sequences filter can be used to filter out these codes and "clean up" the text so that it can be used in standard fashions such as copying and pasting into a word processor. On Unix machines the man (manual) help utility will only allow page-by-page browsing through a file in a forward direction. By piping the man output to a text file, transferring it to a DOS machine, and running it through the Remove ANSI Escape Sequences filter (and the Convert EOL filter - Unix to DOS if desired), a standard DOS editor can be used for browsing through the file, quoting from it, etc.

This filter can be enabled on the command line by using the /DA option.

See also

Remove menu


Remove email headers

 Remove email headers
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter removes the email headers that accompany emails exported to a text format. The email headers are the lines such as To:, From:, Subject: and various other message headers added by all the servers through which your email passes before it gets to its destination.

See also

Extract email addresses

Extract URLs


Remove all

 Remove all
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Remove (Std and Pro version only)

This filter removes all text. Unlike a pattern match filter that matches everything and then throws it away, this filter is far more efficient, especially for large files, as it signals completion back to the input filter so only the first chunk of a multi-gigabyte file will ever get processed.

It is useful in two main situations

1. Inside a subfilter, it prevents any of the subfiltered text from re-entering the text stream. So you could restrict to lines matching a pattern, output the matching lines to a new file, and then remove them.
2. To remove all of the text of a file, then use an Add Header filter with the @fullInputFilename macro to obtain the name of the file.

Note: An Add Left Margin or Add Right Margin filter will not work after a Remove All filter, as they require an actual line to trigger them. Instead, use an Add Header or Add Footer filter.

See also

Remove menu


Find and replace

 Find and replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

TextPipe offers a number of different search and replace types:

· Exact match - the find string is matched exactly - no special characters such as \ are interpreted with special meanings.
· EasyPattern - an easy to use pattern matching language
· Pattern - the find string is a perl-style pattern match expression
· Sounds like - the find string is used for soundex matching. Soundex matching is a method that makes surnames such as Smith, Schmidt and Smythe all appear identical. It is very useful for finding misspellings of names
· Edit distance - Determines the number of 'edits' between the search word and words in the text being processed
· Old Pattern (egrep) - the find string is an egrep-style pattern match expression, which is not as powerful as perl pattern matching
· Brief pattern match - the find string is a pattern match expression in the style of the text editor Brief
· MS Word pattern - the find string is a pattern match expression in the style of Microsoft Word
· Unicode exact match - used for finding text in Unicode files
· Unicode pattern match - for finding Unicode text patterns in Unicode files

· Load a search/replace list from Excel, CSV or Tab
· Search/replace list
· Find an HTML/XML element (e.g. <table...>)
· Find an HTML/XML attribute (e.g. height="...")
· Find between an HTML/XML pair (e.g. <script>...</script>)

See also

· Macros that can be used in Replace With strings
· An Introduction to Pattern Matching
· Example patterns
· Pattern matching reference
· EasyPattern pattern matching
· EasyPattern reference
· Unicode patterns reference


Converting Search/Replaces from other sources

 Converting Search/Replaces from other sources
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

If you have search/replace pairs in Perl script or text files from other Search/Replace packages, you can use a simple procedure to convert them to TextPipe. You can also add a list of search and replace pairs through the user interface.

Edit the file into the following form

/S=<search string>

/R=<replace string>

/S=<search string 2>

/R=<replace string 2>

/S=<search string 3>

/R=<replace string 3>

..etc

You could even use TextPipe to do this for you!

Save this edited script to a file (call it import.txt). Then start TextPipe as follows:

textpipe.exe /z=import.txt

This effectively reads command-line parameters from a file instead of from the command line. Quotes are not required for search/replace strings with embedded spaces.

If you know that the original source uses Regular Expressions, then you can replace the

/S=<search string>

with

/SX=<pattern string>

Typically, regular expression matches are case-sensitive, so you could also use

/SCX=<case-sensitive pattern>

to ensure the original search/replace is unchanged.

This is the exact procedure used to import TextPipe's Jive, Kraut and Elmer Fudd filters from a variety of sources.


Prompt On Replace

 Prompt On Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The prompt on replace dialog is shown when a search filter with "Confirm before replace" checked finds a match.

Found Text

This tab displays the matching text with a highlight, and where possible, also shows surrounding text so that you see the context of the match. If the replacement text is not appropriate, you can edit it. Where the search filter is NOT a regular expression, changes to the replacement text are used as default replacement text for all future matches. A right-click or context menu lets you insert special characters in the replacement text field.

Search Information

This tab shows the search string or pattern, and if the search filter IS a regular expression match, it also shows the contents of the $0-$9,$a-$z variables for the match (depending on the number of bracketed expressions in the pattern match expression).

Yes | Replace this match. Continue prompting for remaining matches
No | Do not replace this match. Continue prompting for remaining matches
Yes (file) | Replace this match, and all others in this file. Resume prompting at the beginning of the next file.
No (file) | Don't replace this match, or any others in this file. Resume prompting at the beginning of the next file.
Yes (job) | Replace this match, and any others in every file.
No (job) | Don't replace this match, or any others in any file.
Cancel | Stops processing
Yes (subset) | When the text has been extracted by a sub filter, two extra buttons are displayed for replacing all in the remainder of the subset, and skipping all in the remainder of the subset.
No (subset) | When the text has been extracted by a sub filter, two extra buttons are displayed for replacing all in the remainder of the subset, and skipping all in the remainder of the subset.

Hex Dump

The hex dump tab displays the raw hex values of the matched text. This can be very useful for examining binary data and Unicode data.


Exact Search and replace

 Exact Search and replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find text that appears exactly as you have typed it, or with a case variation. You can also use this filter to find multi-line text if you press [Enter] between each line.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

Find String

The text to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy. NO unprintable or special characters can be entered - only what can be pasted or typed on the keyboard.

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Only match the Find String when it is not part of a larger word eg find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags
Send variable X to subfilter (X varies from 1 to 9) | Captured variable X (from 1 to 9) is sent to the subfilter for further processing. This is very handy for detecting text based on variable data stored around it, while leaving the variable data alone. It also avoids the need to use look ahead or look behind assertions in the pattern matching language, which are quite limited and counter-intuitive (hey, we didn't write that bit)

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Perl Pattern Search and Replace

 Perl Pattern Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find a text pattern. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find pattern (perl-style)

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters and inserting pattern match characters.

Note: The special character '$' (end of line) may not work as expected because the pattern library we use expects Unix end of lines. Use the expression [\r\n] instead.

To capture data, simply enclose it in parentheses e.g. '(\d+?)' captures the longest string of digits to variable $1. To capture data and also name it, use this syntax '(?P<digits>\d+?)'. This makes the digits available both in $1 and in the global macro @digits.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

See also

· An Introduction to Pattern Matching
· Example patterns
· Pattern matching reference

Pattern options button [...]

Default Match Length ("Greediness")

When set to maximal match, TextPipe will try to match the pattern against the longest piece of text possible. When set to minimal match, TextPipe will try to match the pattern against the shortest piece of text possible. Minimal matching is the default and is the most often used, because it prevents the .* operator from matching an end tag at the end of the file rather than the very next one, and for this reason it is generally much faster than a maximal match. With a large maximum text buffer size, greedy matching can be very inefficient.

This option inverts the "greediness" of the quantifiers so that they are not greedy by default, but become greedy if followed by "?". It can also be set by a (?U) option setting within the pattern.

Maximum Text Buffer Size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Technical Notes:

The pattern matching engine is given a block of text to work with. This block of text is guaranteed by TextPipe to be at least 4K larger than the size of the "Maximum Text Buffer Size" - the block may be larger depending on what filters precede the pattern match filter. Let's call this buffer size the 'critical size'. This critical size allows for efficient matching, because if the buffer is too large, the pattern matching engine may have to check many cases before it fails, so it is more efficient to make it fail earlier by having a smaller buffer size. If greedy matching is turned on, having a larger buffer size may make matching very inefficient.

Allow Comments In Pattern

This enables Perl-style comments and extra white space to be included the pattern match. White space data characters in the pattern are totally ignored except when escaped or inside a character class, and characters between an unescaped # outside a character class and the next new line character, inclusive, are also ignored. This is very handy for including comments inside complicated patterns. Note, however, that this applies only to data characters. White space characters may never appear within special character sequences in a pattern, for example within the sequence (?( which introduces a conditional sub pattern.

'.' Matches Newline

When enabled (default) a dot metacharacter in the pattern matches all characters, including new lines. Without it, new lines are excluded. A negative class such as [^a] always matches a new line character, independent of the setting of this option. A Newline is both ASCII 10 and ASCII 13.

Enable UTF-8 Support

Enables pattern matching in UTF-8 encoded data.

Note: If you search for the Unicode character

\x{2014}

the matching engine replaces this with \xE2\x80\x94 internally. However, if we specify a search term of \xE2\x80\x94 with the utf8 option ON, the matching engine will not find anything. If we use \xE2\x80\x94 with utf8 UNCHECKED, then it will find it as normal.

Note: Case sensitive replace may give unexpected results for UTF-8 patterns.

Match Case

When checked, the case of the string found must match the case of the Find Pattern exactly.

Find Whole Words Only

Only match the Find Pattern when it is not part of a larger word e.g. find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

$1 - is replaced with the first parenthesised expression.

$2 - is replaced with the second parenthesised expression

$3 - $9, $10, $11, $12...

$a - $z are synonyms for $10 - $36.

Note: Named captured variables are stored in global macros e.g. @mycapturedvalue.

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note 1: If you capture data to named variables and then reference them in the Replace With string (e.g. @first_name), this can be 2-3.5 times SLOWER than specifying the captured expression number (e.g. $3$).

Note 2: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags
Send variable X to subfilter (X varies from 1 to 9) | Captured variable X (from 1 to 9) is sent to the subfilter for further processing. This is very handy for detecting text based on variable data stored around it, while leaving the variable data alone. It also avoids the need to use look ahead or look behind assertions in the pattern matching language, which are quite limited and counter-intuitive (hey, we didn't write that bit)

Case Sensitive Replace

If the Find Pattern matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find Pattern matches text that is entirely in lowercase, this forces the replace string to all lowercase. Note: Case sensitive replace may give unexpected results for UTF-8 patterns.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


EasyPattern Search and Replace

 EasyPattern Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find a text pattern. You can also use this filter to find multi-line text. This filter uses the EasyPattern pattern matching language, which is very easy to use.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find EasyPattern

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

See also

· EasyPattern pattern matching
· EasyPattern reference

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Match Case

When checked, the case of the string found must match the case of the Find Pattern exactly.

Find Whole Words Only

Only match the Find Pattern when it is not part of a larger word e.g. find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

$1 - is replaced with the first parenthesised expression.

$2 - is replaced with the second parenthesised expression

$3 - $9, $10, $11, $12...

$a - $z are synonyms for $10 - $36.

Note: Named captured variables are stored in global macros e.g. @mycapturedvalue.

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note 1: If you capture data to named variables and then reference them in the Replace With string (e.g. @first_name), this can be 2-3.5 times SLOWER than specifying the captured expression number (e.g. $3$).

Note 2: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags
Send variable X to subfilter (X varies from 1 to 9) | Captured variable X (from 1 to 9) is sent to the subfilter for further processing. This is very handy for detecting text based on variable data stored around it, while leaving the variable data alone. It also avoids the need to use look ahead or look behind assertions in the pattern matching language, which are quite limited and counter-intuitive (hey, we didn't write that bit)

Case Sensitive Replace

If the Find Pattern matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find Pattern matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Sounds-like Search and Replace

 Sounds-like Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find a word that sounds like another word, usually to locate a spelling mistake or a spelling variation. Sounds-like (soundex) matching makes surnames such as Smith, Schmidt and Smythe all appear identical.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find a word that sounds like

The single word to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy. No special characters are allowed.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

Match Case

Disabled. All words are compared without case sensitivity.

Find Whole Words Only

Disabled - always on by default.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Edit Distance Search and Replace

 Edit Distance Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find a word that is spelt like another word, usually to locate a spelling mistake or a spelling variation.

An edit distance counts the minimum number of 'edits' required to change one word into another. A count of 1 is given to each character delete, character insert or swap of two adjacent characters. Extra deletes, inserts or swaps increase the edit distance between two words e.g.

Word 1 | Word 2 | Edit distance
Delphi | Delhi | 1 (delete)
Delhpi | Delphi | 1 (swap)
Dlphi | Delphi | 1 (insert)
Dlhpi | Delphi | 2 (insert + swap)

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find a word similar to

The single word to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy. No special characters are allowed.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

Pattern options button [...]

Changes the edit distance threshold. Any strings with an edit distance less than this value will be replaced by the replace string.

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Disabled - always on by default.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Old Pattern (egrep) Search and Replace

 Old Pattern (egrep) Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find a text pattern. You can also use this filter to find multi-line text. This filter is slower and less flexible than using a perl pattern, but is maintained for backward compatibility.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Find pattern (old egrep style)

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters and inserting pattern match characters.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

See also

· An Introduction to Pattern Matching
· Example patterns
· Pattern matching reference Note: This pattern type does NOT support perl-style {...} quantifiers, or perl-style (?:....) brackets.

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Match Case

When checked, the case of the string found must match the case of the Find Pattern exactly.

Find Whole Words Only

Only match the Find Pattern when it is not part of a larger word e.g. find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

$1 - is replaced with the first parenthesised expression.

$2 - is replaced with the second parenthesised expression

$3 - $9, $a - $z

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags
Send variable X to subfilter (X varies from 1 to 9) | Captured variable X (from 1 to 9) is sent to the subfilter for further processing. This is very handy for detecting text based on variable data stored around it, while leaving the variable data alone. It also avoids the need to use look ahead or look behind assertions in the pattern matching language, which are quite limited and counter-intuitive (hey, we didn't write that bit)

Case Sensitive Replace

If the Find Pattern matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find Pattern matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Brief Pattern Search and Replace

 Brief Pattern Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

This filter is identical in function to a pattern search and replace, except that it supports the Brief-editor syntax for specifying patterns:

Brief | TextPipe/perl pattern | Function
? | . | Match any character
@ | .* | Match zero or more characters
+ | .+ | Match one or more characters
< | ^ | Match the start of line
% | ^ | Match the start of line
$ | $ | Match the end of line
> | $ | Match the end of line
{ | ( | Start a group
} | ) | End a group


MS Word Pattern Search and Replace

 MS Word Pattern Search and Replace
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

This filter is identical in function to a pattern search and replace, except that it supports MS Word-style syntax for specifying patterns:

To find | Use this wildcard | Examples
Any single character | ? | s?t finds "sat" and "set".
Any string of characters | * | s*d finds "sad" and "started".
One of the specified characters | [ ] | w[io]n finds "win" and "won".
Any single character in this range | [-] | [r-t]ight finds "right" and "sight". Ranges must be in ascending order.
Any single character except the characters inside the brackets | [!] | m[!a]st finds "mist" and "most", but not "mast".
Any single character except characters in the range inside the brackets | [!x-z] | t[!a-m]ck finds "tock" and "tuck", but not "tack" or "tick".
Exactly n occurrences of the previous character or expression | {n} | fe{2}d finds "feed" but not "fed".
At least n occurrences of the previous character or expression | {n,} | fe{1,}d finds "fed" and "feed".
From n to m occurrences of the previous character or expression | {n,m} | 10{1,3} finds "10", "100", and "1000".
One or more occurrences of the previous character or expression | @ | lo@t finds "lot" and "loot".
The beginning of a word | < | <(inter) finds "interesting" and "intercept", but not "splintered".
The end of a word | > | (in)> finds "in" and "within", but not "interesting".

Notes

· Type <(pre)*(ed)> to find "presorted" and "prevented".

· You can use the $n wildcard to search for an expression and then replace it with the rearranged expression. For example, type (Newton) (Christie) in the Find what box and $\2 $1 in the Replace with box. TextPipe will find "Newton Christie" and replace it with "Christie Newton".

MS Word Pattern Troubleshooting

Type the search text correctly Make sure you didn't mistype the text or add extra spaces before or after it.

Clear the Match case check box If this check box is selected, TextPipe finds only text with the specified capitalization (for example, if you search for "Most", TextPipe won't find "most" or "MOST").

Clear the Find whole words only check box If this check box is selected, TextPipe finds only complete words (for example, if you search for "enter", TextPipe won't find "entertain").

When I use MS Word wildcard characters, TextPipe doesn't find certain text.

Examples For example, a search for s*t will find "sat" but not "Sat" or "SAT". If you want to search for any combination of uppercase and lowercase letters, use the square brackets wildcard. For example, type [Ss]*[Tt] to find "sat", "Sat", or "SAT".

Specify all variations of a word When you use wildcards, TextPipe finds whole words only. For example, a search for e*r will find "enter" but not "entertain". (Notice that the Find whole words only check box is dimmed to indicate that this option is automatically on; you can't turn off the option.) If you want to search for all variations of a word, use the appropriate wildcard characters. For example, type <(e*r) to find "enter" and "entertain".


Unicode Search and Replace (UTF-16LE)

 Unicode Search and Replace (UTF-16LE)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find Unicode text that appears exactly as you have typed it, or with a case variation. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Note 1: The Unicode file being processed must be in UTF-16LE form. If it is not, use the Unicode conversion filter to convert it.

Note 2: If you're using the Trial Run area to test the replacement, ensure that the Trial Run Extra Options are set to Treat Trial Input as Unicode (checked) and Treat Trial Output as Unicode (checked).

Find Unicode String

The text to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Ignore non-spacing characters

Ignores characters that have zero width

Space compression

Treats multiple spaces and tabs the same as a single space

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Only match the Find String when it is not part of a larger word eg find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Unicode Pattern Search and Replace (UTF-16LE)

 Unicode Pattern Search and Replace (UTF-16LE)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

Use this filter to find Unicode text that appears exactly as you have typed it, or with a case variation. You can also use this filter to find multi-line text.

The length of each string is shown above it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files.

Notes:

1. The Unicode file being processed must be in UTF-16LE form. If it is not, use the Unicode conversion filter to convert it.
2. If you're using the Trial Run area to test the replacement, ensure that the Trial Run Extra Options are set to Treat Trial Input as Unicode (checked) and Treat Trial Output as Unicode (checked).
3. The regular expression and text must be normalized.
4. Conversion to lower case assumes a 1-1 mapping.
5. Unicode patterns do not support capturing, lookahead, lookbehind, or greedy/non-greedy quantifiers. They are non-greedy by default.

Find Unicode Pattern

The pattern to find. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, and entering unprintable and special characters.

The length of the search string and replace string is shown below it, and a warning is displayed on the status line if they are not the same. This can be very useful when replacing strings in binary files, as the search and replace text need to be the same length to avoid corrupting the file format.

See also

· Unicode patterns reference

Pattern options button [...]

Maximum match size

Changes the maximum search buffer size. If you need to match a string that is longer than 4096 bytes (4k), increase the Maximum text buffer size to be at least as large as the string you need to match.

Ignore non-spacing characters

Ignores characters that have zero width

Space compression

Treats multiple spaces and tabs the same as a single space

Match Case

When checked, the case of the string found must match the case of the Find String exactly.

Find Whole Words Only

Only match the Find String when it is not part of a larger word eg find "the" but not "them", "there", "worth" etc.

Replace With

The text to replace the found text with. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering unprintable and special characters, macros and inserting captured variables:

$0 - is replaced by the entire matched text

Special requirements | Replace With text: | With Captured Variables | Results in:
If you need to put a dollar sign $ in the Replace With section, simply double it i.e. $$. | hello$$ | N/A | hello$
If you need to include the literal name of a variable, just double the $ | $$1 | $1=AAA | $1
If you need to place digits immediately after a captured variable, end the captured variable name with '$'. | $1$16 | $1=AAA, $16=BBB | AAA16
 | $1$$$ | $1=AAA | AAA$
 | $1$$16 | $1=AAA, $16=BBB | AAABBB
Macros can be inserted to provide the value of a global variable | a@randomdigit@a | @randomdigit@=7 | a7a

Note: Older versions of TextPipe only used $0-$9, so there was no requirement to disambiguate $11 and $11. TextPipe detects a file being loaded from an older version and makes this change automatically. If you follow the simple rule of always following a captured variable with a $ sign then you can't go wrong.

The replacement text is passed to any sub filters.

Action

The Action field defines what happens when the text is found.

Action Name | Action Description
Replace | Default. The found text is replaced with the new text
Remove | The found text is removed
Send matching text to subfilter | The found text is sent to the subfilter for further processing, such as changing the capitalization
Send non-matching text to subfilter | Non-matching text is sent to the subfilter for further processing, such as performing a replacement outside of HTML tags
Send variable X to subfilter (X varies from 1 to 9) | Captured variable X (from 1 to 9) is sent to the subfilter for further processing. This is very handy for detecting text based on variable data stored around it, while leaving the variable data alone. It also avoids the need to use look ahead or look behind assertions in the pattern matching language, which are quite limited and counter-intuitive (hey, we didn't write that bit)

Case Sensitive Replace

If the Find String matches text that is entirely in UPPERCASE, this forces the replace string to all UPPERCASE. If the Find String matches text that is entirely in lowercase, this forces the replace string to all lowercase.

Replace First Only

Only replace the first occurrence found - skip all remaining matches.

Prompt On Replace

By checking this option, you can manually verify each match before it gets replaced. You can choose to replace or ignore single matches, the remainder of the file or the entire job.

Skip Prompt If Identical

If the find string and replace string are identical, this option can be enabled to skip prompting. This can happen when the case of the search string is identical to the case of the replace string during a case-insensitive search.

Extract matches

When this option is checked all non-matching text will be discarded, leaving only the replacement text. This can be handy for data mining content from web sites or data files.


Unicode Pattern Reference

 Unicode Pattern Reference
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Regular expression search engine for text in UCS2 form taking surrogates into account. This implementation is an improved translation from the URE package written by Mark Leisher who used a variation of the RE->DFA algorithm done by Mark Hopkins.

Assumptions

1. Regular expression and text already normalized
2. Conversion to lower case assumes a 1-1 mapping.

Definitions

Separator - any one of U+2028, U+2029, NL, CR.

Operators

· . - match any character
· * - match zero or more of the last sub expression
· + - match one or more of the last sub expression
· ? - match zero or one of the last sub expression
· () - sub expression grouping
· {m, n} - match at least m occurrences and up to n occurrences

Note: both values can be 0 or omitted which denotes then a unlimited bound {,} and {0,} and {0, 0} correspond to * {, 1} and {0, 1} correspond to ? {1,} and {1, 0} correspond to +

· {m} - match exactly m occurrences

Notes

The "." operator normally does not match separators, but a flag is available that will allow this operator to match a separator.

Literals and Constants

· c - literal UCS2 character
· \x.... - hexadecimal number of up to 4 digits
· \X.... - hexadecimal number of up to 4 digits
· \u.... - hexadecimal number of up to 4 digits
· \U.... - hexadecimal number of up to 4 digits

Character classes

· [...] - Character class
· [^...] - Negated character class
· \pN1,N2,...,Nn - Character properties class
· \PN1,N2,...,Nn - Negated character properties class

POSIX character classes recognized: :alnum: :alpha: :cntrl: :digit: :graph: :lower: :print: :punct: :space: :upper: :xdigit:

Notes

· Character property classes are \p or \P followed by a comma separated list of integers between 0 and the maximum entry index in TCharacterCategory. These integers directly correspond to the TCharacterCategory enumeration entries below. Note: upper, lower and title case classes need to have case sensitive search be enabled to match correctly!
· Character classes can contain literals, constants and character property classes. Example:

[abc\U10A\p0,13,4]

TCharacterCategory

TCharacterCategory = (

// normative categories

0- ccLetterUppercase,

1 - ccLetterLowercase,

2 - ccLetterTitlecase,

3 - ccMarkNonSpacing,

4 - ccMarkSpacingCombining,

5 - ccMarkEnclosing,

6 - ccNumberDecimalDigit,

7 - ccNumberLetter,

8 - ccNumberOther,

9 - ccSeparatorSpace,

10 - ccSeparatorLine,

11 - ccSeparatorParagraph,

12 - ccOtherControl,

13 - ccOtherFormat,

14 - ccOtherSurrogate,

15 - ccOtherPrivate,

16 - ccOtherUnassigned,

// informative categories

17 - ccLetterModifier,

18 - ccLetterOther,

19 - ccPunctuationConnector,

20 - ccPunctuationDash,

21 - ccPunctuationOpen,

22 - ccPunctuationClose,

23 - ccPunctuationInitialQuote,

24 - ccPunctuationFinalQuote,

25 - ccPunctuationOther,

26 - ccSymbolMath,

27 - ccSymbolCurrency,

28 - ccSymbolModifier,

29 - ccSymbolOther,

// bidirectional categories

30 - ccLeftToRight,

31 - ccLeftToRightEmbedding,

32 - ccLeftToRightOverride,

33 - ccRightToLeft,

34 - ccRightToLeftArabic,

35 - ccRightToLeftEmbedding,

36 - ccRightToLeftOverride,

37 - ccPopDirectionalFormat,

38 - ccEuropeanNumber,

39 - ccEuropeanNumberSeparator,

40 - ccEuropeanNumberTerminator,

41 - ccArabicNumber,

42 - ccCommonNumberSeparator,

43 - ccBoundaryNeutral,

44 - ccSegmentSeparator, // this includes tab and vertical tab

45 - ccWhiteSpace,

46 - ccOtherNeutrals,

// self defined categories, they do not appear in the Unicode data file

47 - ccComposed, // can be decomposed

48 - ccNonBreaking,

49 - ccSymmetric, // has left and right forms

50 - ccHexDigit,

51 - ccQuotationMark,

52 - ccMirroring,

53 - ccSpaceOther,

54 - ccAssigned // means there is a definition in the Unicode standard

);


Find inside HTML

 Find inside HTML
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict

This filter can be used to restrict a sub filter to just a small fragment of HTML or XML.

Restriction Type | Example | Possible use
Restrict to tag <table> | <table height="10" width="20"> .... </table> |
- with Include the entire tag checked | <table height="10" width="20"> .... </table> | Just add a Remove All sub filter to remove the tag entirely
Restrict to attribute 'height' | <table height="10" width="20"> .... </table> |
- with Include the quotes checked | <table height="10" width="20"> .... </table> |
- with Include the attribute name checked | <table height="10" width="20"> .... </table> | Just add a Remove All sub filter to remove the attribute entirely
Restrict to between tags <table> and </table> | <table height="10" width="20"> .... </table> |
- with Include the start and end tag checked | <table height="10" width="20"> .... </table> | Just add a Remove All sub filter to remove the tag entirely

If your tag or attribute is not in the list, just type it in.

Note:

· This filter copes with single and double quotes, and escaped quotes (either doubled or escaped with backslash)
· This filter copes with angle brackets within single and double quotes
· <!-- --> comment tags are handled specially. Comment tags can have ANY text inside them - quotes are not matched
· <!DOCTYPE > tags are handled specially to allow for included DTDs enclosed in []
· <!CDATA[ ... ]]> tags are handled specially to allow for any text within the square brackets
· This filter will match a maximum of 4KB of text, so if your tags have more than 4KB of text between them you'll have to use a search/replace filter as the restriction instead.

Type 1. Restrict to tag

This filter restricts sub filters to operate on just the text of a single tag. If your tag is not in the list, just type it in.

Include the entire tag

When checked, the entire tag is passed to the sub filter, not just the text after the tag name. Normally this is not required as the existing tag is being edited. If you want to remove the entire tag, then check this field, and add a Remove All sub filter.

Type 2. Restrict to attribute

This filter restricts sub filters to operate on just the text within the given tag. Normally this filter is inside a Restrict to tag filter e.g. 'Restrict to tag <table...>', sub filter of 'Restrict to attribute height="..."', sub filter of search and replace. If your attribute is not in the list, just type it in. Both double quote chr(22) and the apostrophe chr(27) can be used around the attribute value, and values without white space will be found even without quotes e.g. height=22.

Include the quotes

When checked, the surrounding single or double quotes (if any) are also sent to the sub filter. Normally they are not required as the existing value is being edited. If you want to remove the entire attribute, then check this field and the next field, and add a Remove All sub filter.

Include the attribute name

When checked, the actual attribute name and the equals sign are also sent to the sub filter. Normally they are not required as the existing value is being edited. If you want to remove the entire attribute, then check this field and the previous field, and add a Remove All sub filter.

Type 3. Restrict to between tags

This filter restricts sub filters to operate on just the text between the given tag pairs (which need not match). If your tag is not in the list, just type it in.

Include the start and end tag

When checked, the actual start and end tags are also sent to the sub filter. Normally they are not required as the existing value is being edited. If you want to remove the start and end tag and all text between them, then check this field, and add a Remove All sub filter.

Options

The following options apply to ALL the restriction types above:

Match case

When checked, the tag name must match the case of the name exactly. Normally this is checked for XML, which is case-sensitive, and unchecked for HTML, which is not case-sensitive.

Maximum match size

If you find that the filter is not matching all occurrences of the HTML or XML you're trying to find, increase the maximum match size so that it is larger than the maximum amount of text you expect to find. The default value is 32768 bytes (32K).

See also

Remove duplicate lines

Count duplicates

Randomize lines

Restriction filters


Search/replace list

 Search/replace list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

This filter is used to easily define a set of search/replace pairs that have identical search options.

Only log when a replace occurs - this option is designed not to flood the log when there are thousands of non-matching entries. When logging is turned on and this option is checked, log entries are only made if there have been one or more replacements.

Replacement list

You can specify either a file to load or use the grid to specify multiple search/replace pairs.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path e.g. file.txt or .\output\file.txt or ..\output\file.txt.

You can load Excel data from .XLS files, Tab-delimited data from .TAB files, any other file type (including .CSV) is expected to be in Comma-delimited format. Note - despite Microsoft Excel 2007 and later being able to handle 1 million rows, it is unable to provide data beyond 65535 rows to TextPipe. To work around this, save your Microsoft Excel data to a .CSV or .TAB temporary file so that TextPipe can import all the data.

Using these values

Use the grid to specify multiple search/replace pairs.

Use the [Copy to Clipboard] and [Paste from Clipboard] buttons to transfer the grid rows to and from the clipboard for editing (in tab-delimited format).

You can re-order rows by selecting them with the row bar and then dragging them to the new position.

The Swap button exchanges Find and Replace strings. This is useful if you encode certain character at the start of a file and reverse that order at the end.

The Reverse button reverses the order that rows are applied. This is used in conjunction with Swap.

Process Longest 'Find' strings first - checking this option searches for long phrases (most specific) before short phrases (least specific) - this is generally used for translations.

Simultaneous search - checking this option results in all search strings being scanned for simultaneously instead of consecutively. This is very useful if the search strings and results strings overlap e.g.

Let’s say the replace list is:

1->7

7->10

10->1

In a consecutive search – each search/replace term gets applied one after the other and the results overlap:

Input text | After first filter (1->7) | After second filter (7->10) | After third filter (10->1) (output text)
1 | 7 | 10 | 1
7 | 7 | 10 | 1
10 | 70 | 100 | 10

In a simultaneous search – each search term is examined in parallel:

Input text | Output text
1 | 7
7 | 10
10 | 1

Exact match - searches will be combined into one matrix for ultra-fast searching of 100,000+ terms

Other match types - Patterns are now allowed - depending on length, approx 100 terms are grouped together to combine terms.

There are two types of exception that you need to think about when replacing simultaneously.

Exception 1. Duplicate search phrases

Imagine the replace list

aa->bb
bb->dd
aa->cc

With input text of:

aa bb

The output is always

bb dd

- the second replacement of aa->cc is ignored (TextPipe issues a warning for this). Note that if the search/replaces were performed consecutively, the result would be

dd dd

The string can contain unprintable characters. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

Find/Replace options

Find type, Match case etc, as for normal search/replace.

From file

Loads a series of search/replace pairs from a file dynamically at run time. If the file extension is .XLS or .XLSX, the file is assumed to be Excel format, if the extension is .TAB the file is assumed to have tab-delimited values, and any other extension (including .CSV) is assumed to have Comma-Separated Values.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Excel spreadsheet (.xls)

The Excel spreadsheet must have data in Sheet 1, and be formatted with the search term in column A, and the replacement for it in column B. No heading is allowed. The loading process stops when a blank cell is reached in column A. It might look as follows:

Tab-delimited file (.xls)

The list is loaded from a Tab-delimited file. Such a file can be created with Notepad, and looks like this:

a       b

aaaabbb cccccdddd ssss

a

a

a       b      c

CSV-delimited file (.csv or any other extension)

The list is loaded from a CSV-delimited file. Such a file can be created with Notepad, and looks like this:

a,b

"a","b"

"aaaa""bbb","ccccc""dddd""ssss"

a,

a

a,b,c

"a","b","c"

"aaaa""bbb","ccccc""dddd""ssss","sssss""ssss"


Load a search/replace list

 Load a search/replace list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Replace

This series of menu options allows you to import a list or table of search and replace pairs from an external file. The type of search and replace added, and the options that are set, are predetermined by the last search and replace filter selected. So to set the default options, you must initially select a search and replace filter with the desired options. This can also be done programmatically using the COM interface, or search and replace pairs can be added via the command line using pairs of /S= and /R= options. You can also use a Search/Replace list filter to load a file dynamically.

Import list from Excel

The Excel spreadsheet must have data in Sheet 1, and be formatted with the search term in column A, and the replacement for it in column B. No heading is allowed. The loading process stops when a blank cell is reached in column A. It might look as follows:

Import list from CSV-delimited file

The list is loaded from a CSV-delimited file. Such a file can be created with Notepad, and looks like this:

a,b

"a","b"

"aaaa""bbb","ccccc""dddd""ssss"

a,

a

a,b,c

"a","b","c"

"aaaa""bbb","ccccc""dddd""ssss","sssss""ssss"

Import list from Tab-delimited file

The list is loaded from a Tab-delimited file. Such a file can be created with Notepad, and looks like this:

a       b

aaaabbb cccccdddd ssss

a

a

a       b      c


Extract Filters

 Extract Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

Extract matches - same as Search/replace but with Extract option turned on

Extract matches by line (Pro version only)

Extract email addresses

Extract URLs

Extract matching lines (grep)

Extract non-matching lines (inverse grep)

Extract lines from list (Pro version only)

Extract lines matching pattern list (Pro version only)


Extract lines matching (grep)/non-matching (inverse grep)

 Extract lines matching (grep)/non-matching (inverse grep)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

TextPipe will retain any lines that match/don't match the given pattern, all other lines will be deleted. For options, please see the Restrict to matching lines filter. Unlike that filter, this filter is not allowed to have any sub filters.

Note - any non-matching lines are discarded, so you can't really use two Retain Matching Lines filters in a row. Instead, use an or '|' operator in the pattern match.

Note 2 - this filter will not match blank lines.

See also

Special Menu

EasyPattern matching

Perl pattern matching

Remove matching/non-matching lines

Extract lines matching pattern list


Extract email addresses

 Extract email addresses
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

Extract email addresses. This filter searches for email addresses of the form person@server.domain, and writes them out one per line (using a DOS line feed, CR/LF).

Usually this filter is followed by a Sort filter, with the remove duplicate lines option. This is followed by a Search and Replace filter, searching for \013\010 and replacing with a comma or semi-colon, depending on the email address separator used by your email software.

See also

Remove email headers

Extract URLs


Extract URLs

 Extract URLs
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

Extract URLs. This filter lists http://, https://, ftp:// and gopher:// URLs one per line (using a DOS line feed, CR/LF).

Usually this filter is followed by a Sort filter, with the remove duplicate lines option.

See also

Extract email addresses

Remove email headers


Extract matches

 Extract matches
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

This filter keeps text (not lines) that match a regular expression, and outputs each match with a CR/LF between each match. All other text is discarded. If you do not need this filters ability to also include the current filename and line number, you should use the Extract option of the Find and Replace filter because it is far more flexible.

Extract Matches

The string or pattern to find. The string can be found anywhere on a line or over multiple lines. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

Ignore Case

When checked, the case of the string found doesn't have to be identical to the Match Pattern exactly.

Output Count of Matches

When checked, outputs a count of the number of matches in each file.

Include Filename

When checked, outputs the filename prior to showing its matches.

See also

Retain matching lines

Remove matching lines


Extract lines from list

 Extract lines from list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

TextPipe will retain all lines that EXACTLY match any of the lines in the given text file, or in the text memo area, all other lines will be deleted (line breaks can be of any format).

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Special Menu

Extract lines matching pattern list


Extract lines matching pattern list

 Extract lines matching pattern list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Extract (Std and Pro version only)

TextPipe will Extract all lines that match any of the patterns in the given text file (or specified in the text memo area), all other lines will be discarded. Each pattern must be specified on a line of its own (line breaks can be of any format). Each line is interpreted as a perl-style pattern, with options of case-insensitive matching, non-greedy matching, comments not allowed. If you need to change these options, precede each line by the following codes:

Change option command | Result
(?-i) | Case-sensitive matching
(?x) | Comments allowed
(?U) | Greedy matching

The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting macros and global variables.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Extract matching lines

Extract non-matching lines

Extract lines from list

Extract lines matching pattern list

Remove menu


Special Filters

 Special Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special

These filters provide specialized processing suited to a variety of tasks.

Extract matches by line - now in Extract menu (Pro version only)

Database connection (Pro version only)

Sort

Run external program (Pro version only)

Script (Pro version only)

Capture text (Pro version only)

Comment

Debug filter

XSLT filter

Shred file

Switch CPU

Split files

Merge (join) files

Count duplicate lines

Move columns (Pro version only)

Move fields (CSV, Tab, Pipe etc) (Pro version only)

Copy columns (Pro version only)

Copy fields (CSV, Tab, Pipe etc) (Pro version only)

Repeat Subfilters X Times

Repeat file contents

Randomize lines

Reverse each line

Reverse line order


Database connection

 Database connection
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The database connection filter connects to an OLE DB/ODBC database and retrieves data (one or more record sets if data is retrieved from a stored procedure). It assumes that its input text is a sequence of SQL statements that can be run against the specified database. The data (if any) returned after executing each statement is formatted according to the output style. The filter assumes that the input text is a series of semi-colon (;) terminated SQL statements. Semi-colons that are inside single or double quotes are ignored.

The data output can then be manipulated by subsequent filters to

· generate a database extract (which can be converted, modified or reformatted)
· update data from the original table after performing name and address cleansing (requires a subsequent database connection filter)
· insert data into a new table (requires a subsequent database connection filter)

Sample extracting from a database

Here's a quick example of how to use a database filter to extract data from a database:

1. Add a Special Menu\Database filter
2. Configure it to point to your database using the Build button e.g. to the Northwind database that comes with MS Access
3. Set the database filter output options, e.g. for Delimited Output
4. In the Trial Run area, type

select * from customers

or use

select * from <your table name>

1. Click the [Trial Run] button, to see data extracted and populated in the Trial Run Output area. You can also set the Input Filter to use the Trial Run area by default, and then just click the [Go] button. To save this data to a file, set the Single File Output option of the Output Filter.

Sample extract from Excel workbook

Using the same steps above, for step 2, first create a System DSN (Control Panel\Data Sources\System DSN) pointing at your Excel spreadsheet. Set TextPipe's database filter at this new ODBC connection. Then use

SELECT * FROM [Sheet1$]

in the Trial Run input area

Sample Insert into Access DB

Using the same steps above, use the following Trial Run Input text for a table called 'Address':

INSERT INTO Person (first,last,age) values ('Simon', 'Carter', 72);

INSERT INTO Person (first,last,age) values ('Joe', 'Blogs', 16);

Ensure multiple INSERT statements have a semi-colon (;) between them. Also note the some databases, most notably Microsoft Access, are extremely finicky about which quotes you use. MS Access will ONLY accept single quotes around strings - double quotes lead to spurious errors. Dates are usually entered like a string, with quotes around them.

Database connection string

The database connection string gives TextPipe the name of the database and tells it how to connect to it. We recommend using the Build button unless you are familiar with setting the connection string manually.

Build

The build button allows the connection information to be constructed. A username and password can be specified in the database connection string if required.

Command Timeout

The command timeout (in seconds), allows you to set the time after which a query will be automatically failed. A setting of 30 will abort a query with an error message if it runs for longer than 30 seconds (the default). The command timeout setting is not implemented by all database drivers. The maximum value is 2147483647.

Prepare query

When checked, the query is prepared (pre-compiled) before execution by the database driver. This can make it faster to execute, however not all queries can be prepared (such as those with aliases etc) - this is database independent. If you have compatibility problems please uncheck this option.

Output Style

The output style determines how the data will be output:

Output header row/DTD

When enabled, an XML document type definition (DTD) is output for XML files, or a header line with column titles is output for Delimited or Fixed width output formats. TextPipe forces all column heading to be unique, and to conform to XML naming conventions without any special characters. If TextPipe doesn't automatically generate the header you need to can easily add your own using an Add Header filter.

Delimited

The field delimiter appears between each column of data, and the text qualifier appears at the start and end of string or character type fields. If the string field contains the delimiter character it is escaped by repeating the delimiter character. A tab character (\t or \009) may be used to separate columns to generate tab delimited data. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters. If Output Header Row/DTD is checked TextPipe automatically generates a Header Row with column names (shown in italics at the start of the example below).

"company","productid"

"Dushkin/MCgraw-Hill",100503

"Programma Electric AB",131686

"Ohio University",131686

"DSL Shops",131686

Fixed

The data is output left justified in a column width determined by the database (specific to each field). If the column width is greater than 4096 then the field is output with no padding. If Output Header Row/DTD is checked TextPipe automatically generates a Header Row with column names (shown in italics at the start of the example below).

company                    productid

Dushkin/MCgraw-Hill        100503

Programma Electric AB      131686

Ohio University            131686

DSL Shops                  131686

XML

TextPipe outputs the data in XML format, with each column output an an entity. All XML-special characters are escaped. If Output Header Row/DTD is checked TextPipe automatically generates an XML schema (DTD) for the data being extracted (shown in italics at the start and end of the example below). You can use an Add Header filter to add your own DTD at the start of the file.

<?xml version="1.0" standalone="yes" ?>

  <!-- generator="TextPipe Search/Replace" -->

  <!-- companyURL="www.datamystic.com" -->

  <!-- SQLquery="select company,productid from shareit" -->

  <!-- event="DTD start" -->

  <!DOCTYPE TextPipeRows [

    <!ELEMENT TextPipeRows (TextPipeRow*)>

    <!ELEMENT TextPipeRow (company, productid)>

    <!ELEMENT company (#PCDATA)>

    <!ELEMENT productid (#PCDATA)>

  ]>

  <!-- event="DTD end" -->

  <TextPipeRows>

  <TextPipeRow>

    <company>Dushkin/MCgraw-Hill</company>

    <productid>100503</productid>

  </TextPipeRow>

  <TextPipeRow>

    <company>Programma Electric AB</company>

    <productid>131686</productid>

  </TextPipeRow>

  <TextPipeRow>

    <company>Ohio University</company>

    <productid>131686</productid>

  </TextPipeRow>

  <TextPipeRow>

    <company>DSL Shops</company>

    <productid>131686</productid>

  </TextPipeRow>

</TextPipeRows>

  <!-- rowCount="4" -->

Insert script for table <tablename>

With this mode setting TextPipe will generate a SQL Insert Script that can be applied to any database. You must specify the name of an existing table to insert the data into. If Output Header Row/DTD is checked, TextPipe includes the column names to insert into (shown in italics in the example below). You can use an Add Header filter to add a drop/create table statement at the start of the script.

insert into existing_table ( company, productid ) values ( "Dushkin/MCgraw-Hill", 100503 );

insert into existing_table ( company, productid ) values ( "Programma Electric AB", 131686 );

insert into existing_table ( company, productid ) values ( "Ohio University", 131686 );

insert into existing_table ( company, productid ) values ( "DSL Shops", 131686 );

BLOB data

If your query returns BLOB (Binary Large Object) data (SQL Server Binary fields), a sub filter can be used to encode the binary data. The most common method to use is a mime base64 encoding.

Notes and Errors

We cannot SELECT data from bigint, decimal & numeric fields. However, you can work around this with a cast e.g. select cast(bigint as varchar) from tablename. Also note that data from Binary fields often contain ASCII 0's - use a Map to replace these nulls with space or similar.

If you have difficulties using this filter, you may need to update the database drivers and the corresponding system components on your system. Click here to download the Microsoft MDAC (Data Access Components) library.


Inserting data into a database

 Inserting data into a database
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

Using TextPipe you can insert processed data directly into a database without having to run a second program.

The database filter assumes that incoming text is one or more SQL commands, each separated by a semi-colon. To insert data into a database, the text generally looks like this:

insert into products ( product_id, productname ) values ( 188875, 'AbusePipe Single User License' );

insert into products ( product_id, productname ) values ( 146467, 'Annual Renewal' );

insert into products ( product_id, productname ) values ( 139425, 'Annual Renewal (18%)' );

(This was created using a database filter set to generate a SQL Insert Script for table 'products', with Trial Run Input of 'select product_id,productname from products')

In the example above we are inserting into the 'products' table, into the fields product_id and productname.

Let's assume that our data is in CSV (Comma Separated Value) form:

188875,AbusePipe Single User License

146467,Annual Renewal

139425,Annual Renewal (18%)

We need to make some simple changes to it to manipulate it into the SQL Insert Script form.

1. String fields usually need to be placed inside single or double quotes (depending on your database). This can be done using the Restrict to delimited fields (CSV, Tab, Pipe etc), with subfilters of an Add Header and Add Footer filter containing a single or double quote.
2. Use an Add Left Margin filter with text of

insert into products ( product_id, productname ) values (

1. Use an Add Right Margin filter with text of

);

1. Add a database filter for the database the table is in.

Tutorial Index


Sort

 Sort
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special

Arranges the input lines into sequence.

Sort Type

The sort type controls the method by which items are sorted. The available options are:

· ANSI sort (case insensitive)
· ANSI sort (case sensitive) - faster than case insensitive as no case-mapping is performed
· ASCII sort (case insensitive)
· ASCII sort (case sensitive) - faster than case insensitive as no case-mapping is performed
· Numeric sort
· Sort by length of line

An ANSI sort is locale-specific (ie can vary from country to country). Lower case letters generally have lower codes than upper case letters, hence a word will sort in lower case before the same word in upper case e.g. 'Freddo' will appear before 'FREDDO'. This is opposite to an ASCII sort where upper case words appear first.

An ASCII sort is based on the ASCII code (or binary value) of each letter in a word. Upper case letters have lower codes than lower case letters, and hence the word 'HELLO' will appear before the word 'hello'.

Both ANSI and ASCII sorting have a case-sensitive and case-insensitive sort. With case-sensitive sorts, the word 'Volkswagon' is different to the word 'volkswagon'. With a case-insensitive, the word 'BMW' is the same as the word 'bmw'.

Numeric sort allows lines to be sorted according to their numeric value. The numeric value must appear at the start of the line (leading spaces are allowed). The number must be in decimal, and can be in floating point format. Any non-numeric characters after the number are ignored. If the line has no valid numeric value it is given a value of 0. If you have dates formatted in YYYYMMDDHHMMSS order you can easily sort by date based on the value.

Sort by length of line allows lines to be sorted by their length. This can be handy for detecting lines in database extracts which are longer or shorter than they are allowed to be.

Sort by date and time allows lines to be sorted by a date and time value. The value must use the current locale’s date/time format. In the US, this is commonly MM/DD/YY HH:MM:SS format. Specifying AM or PM as part of the time is optional, as are the seconds. Use 24-hour time (7:45 PM is entered as 19:45, for example) if AM or PM is not specified.

Sort by date allows lines to be sorted by a date value. The value must consist of two or three numbers, separated by the character defined by the current's locale's Date Separator. The order for month, day, and year is determined by the current locale--possible combinations are m/d/y, d/m/y, and y/m/d. If the value contains only two numbers, it is interpreted as a date (m/d or d/m) in the current year.

Sort by time allows lines to be sorted by a time value. The value must consist of two or three numbers, separated by the current locale's Time Separator character, optionally followed by an AM or PM indicator. The numbers represent hour, minute, and (optionally) seconds, in that order. If the time is followed by AM or PM, it is assumed to be in 12-hour clock format. If no AM or PM indicator is included, the time is assumed to be in 24-hour clock format.

To sort by suffix (by the end of the word), first Apply a Reverse Each Line filter, then sort, then apply the Reverse Each Line filter again.

Remove duplicate lines

Lines with the same sort value can be kept or removed.

Note that there is no way of knowing which line from a group of lines with the same sort value will actually be kept. eg. with input of 'AAA', 'BBB', 'CCC' in a Sort by length of line, there is no way of knowing which of the 3 lines will actually appear in the output as each has a length of 3 and therefore appears to be the same.

Descending order

Arranges the output in descending order (highest value first). The default (unchecked) is to sort in ascending order (lowest value first).

Start column

The Start Column field allows you to ignore leading characters before a comparison is made. This can be very useful for ignoring line numbers is source code etc.

When the Start Column is 1, and the Length is 4096 or more, the sort is optimized compared to selecting a subset of each line. All lines are compared for their entire width.

It can also be used (in conjunction with an external sort program) to remove duplicates from an unsorted file. First add line numbers to every line. Then use an external sort program on the file, ignoring the line numbers (the first 6 columns). Then use this filter (with Ignore Leading Characters set to 6) to remove duplicate lines. Finally, remove the line numbers using the Remove Columns (remove first 6 columns) filter.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

When the Start Column is 1, and the Length is 4096 or more, the sort is optimized compared to selecting a subset of each line. All lines are compared for their entire width.

Maximum Memory to Use

Specifies the maximum MB of physical memory to use, for both TextPipe and the sorting operation. Specify 0 for Automatic - this uses the maximum available physical memory, minus a 30MB safety margin. This option can be very useful in a resource-limited system such as a shared server, where you have limited memory to work with. TextPipe checks the amount of memory in use approximately every 10,000 lines, which is generally around every 0.5 MB of data.

To determine the maximum memory you can use, use Windows Task Manager (Ctrl+Alt+Del) to watch TextPipe's maximum memory consumption. Use this figure, and decrease by 10MB at a time until Out of Memory errors cease.

Technical Notes

· If you have enough memory, the entire sort is performed in memory for speed. If not, it writes the sorted results so far to a temporary file and then continues. If the Output Filter is set to File Output or Single File Output, any temporary files are written to the same folder as the output file. If the Output Filter is set to Clipboard Output then any temporary files are written to the current folder. All temporary files are removed as the sort progresses.

See also

Remove duplicate lines

Count duplicates

Randomize lines


Run external program

 Run external program
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

This filter allows an external program (from a third party) to be run to complement TextPipe's abilities.

The steps that TextPipe follows for each file are:

1. Write all data to the filename specified by "Input file for external program" (after any preceding filters have have performed their work)
2. When the file has been completely processed, run the external program and wait for it to finish. The external program is assumed to use the filename specified by "Input file for external program" as input, and write its output to the filename specified by "Program's output file"
3. If the external program has a non-zero exit code, TextPipe records this in the error log (and continues processing).
4. The data in the filename specified by "Program's output file" is read in and fed through any remaining filters.

Command line

Enter the command line of the program. This must be the COMPLETE command line, including drive, folder and extension of the program, plus any required parameters. You must use double-quotes around any filename that contains spaces. The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Input file for external program

This is the file that TextPipe writes its "text processed so far" to for the external program to then read and process. The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Program's output file

This is the file that the program must write its results to for TextPipe to retrieve and feed into any remaining filters. The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Remove Output Files

When checked, TextPipe will remove the input and output filename at the end of the job. Leave unchecked if you wish to debug the contents of these files.

Minimize External Application

Minimizes the external application when it runs.


VBScript/JScript

 VBScript/JScript
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

This filter allows you to program your own TextPipe filter in VBScript, JScript, VBA, PerlScript, Python, Lisp, REXX etc. You could use your own code to perform custom text processing, e.g. adding line numbers to the end of each line, deleting every second line, deleting every third blank line etc. You could use a message box to ask the user for some input, and then add that input to every line. Or you could invoke Excel, paste text into it, perform a complex statistical calculation and then put the result back in the file. See the example filters in the VBScript folder (such as the one to copy numeric data into an excel spreadsheet, create a 3d graph and then rotate it!). The possibilities are endless!

For the latest information about the extra downloads required, see www.datamystic.com/otherdownloads.html. To make use of this control with Windows Vista you may have to disable User Access Control (UAC).

Inside an script, TextPipe provides the TextPipe internal automation object so that scripts can determine vital data about the current filter, and turn on and off sub filters according to programmed criteria.

See also Automating TextPipe from another program

Script Language

The script language to use. Default values are VBScript and JScript, but you can download and install others (they will not appear in the list - you must enter them manually). TextPipe will display an error message if the language you specified is not installed. You can download the Microsoft VBScript and JScript engines and reference documentation from the Microsoft web site at http://msdn.microsoft.com/scripting/default.htm

Timeout (milliseconds)

Specify the timeout in milliseconds. A script will be aborted if it runs for more than the specified number of milliseconds. The default value is 10000 (i.e. 10 seconds).

Check script

This button checks the script for any errors. If there is an error, a dialog box will appear with a description of the error.

Use default

This button replaces the existing script with the default script for the current language. This can be used as a template for your own code. TextPipe comes with a default script for VBScript inbuilt, and a sample JScript script can be loaded from the VBScript folder.

Set default

This button sets the default script for the current language to the current script. The default script is language dependant, so you can have different scripts for JScript and VBScript etc.

Functions

The following functions are REQUIRED by the script filter. Each filter is followed by explanatory text.

A processing run results in a series of calls as follows:

startJob()

for each input file

startFile()

for each line of the file

   processLine(...)

endFile()

endJob()

function processLine( line : string; EOL : string) : string

TextPipe calls this function for every line in the input file (if your file has 10,000 lines, this function will get called 10,000 times). The function must return the processed line with the EOL string already appended to it.

Lines can be of unlimited length, and are delimited by CR/LF, CR or LF, depending on how the file was generated. TextPipe allows a file to contain more than one kind of end of line character. If needed, you can force all end of line characters to be standard by preceding this filter with a convert end of lines filter. You can also use your own end of line character instead of EOL.

If you need to process multiple lines, paragraphs, pages of text or the entire file, the best approach is to accumulate each line and EOL into a global buffer variable, and then output the entire processed buffer in the endFile() function.

sub startJob()

Called once at the start of a processing job, before any files are processed. Use it to initialize any job-wide variables. A processing job is the combination of all filters applied at once.

sub endJob()

Called once at the end of a processing job, after all files have been processed. Use it to dispose of OLE objects etc. A processing job is the combination of all filters applied at once.

function startFile() : string

Called at the start of each file. Use it to initialize file variables. The return value is any string that must be output at the beginning of the file (header text).

Note: startFile() is ALSO called when the Script is a sub filter, for each text value that the sub filter operates on. The impact of this is that real per-file initialization/finalization needs to be performed in a script that is not inside any sub filters. Global variables set using TextPipe.setGlobalVar should be initialized in this script as well.

function endFile() : string

Called at the end of each file. The return value is any string that must be output at the end of the file (footer text). This may also be used to flush any pending output, such as when the entire file is buffered before being output.

Note: endFile() is ALSO called when the Script is a sub filter, for each text value that the sub filter operates on. The impact of this is that real per-file initialization/finalization needs to be performed in a script that is not inside any sub filters. Global variables set using TextPipe.setGlobalVar should be initialized in this script as well.

Global variables

You can declare variables at the top of the script (see the 'a' variable below). You should initialize these values in the startJob() function, and dispose of them (e.g. for OLE Objects) in the endJob() function. VAriables defined at the top of the script can be used anywhere in the script, but they are invisible to other script filters. To share data between scripts, use global variables, defined use TextPipe.setGlobalVar( name, '...' ) and TextPipe.getGlobalVar( name ).

Debugging

Currently, the easiest way to debug the script is to use VBScript MsgBox() or alert() functions to display text.

Dim MyVar

MyVar = MsgBox ("Hello World!", 65, "MsgBox Example")

   ' MyVar contains either 1 or 2, depending on which button is clicked.

JScript: Does NOT have an alert or MsgBox method. These are part of Internet Explorer, not JScript (but strangely they are in VBScript). Use the alert function of the TextPipe object instead.

Example VBScript

'This example script keeps a line counter that

'does not reset for each file

'Our line counter

dim a

'Called for every line in the file

'EOL contains the end of line characters (Unix, DOS or Mac) that must be

'appended to each line

function processLine(line, EOL)

  a = a + 1

  'add the count to the end of each line

  processLine = line & " " & a & EOL

end function

'Called at the start of a processing job -

'perform one-time initialization here

sub startJob()

  a = 0

end sub

'Called at the end of a processing job -

'destroy any declared objects here

sub endJob()

  'do nothing

end sub

'Called before each file is opened -

'perform per-file initialization here

function startFile()

  startFile = ""

end function

'Called before each file is closed -

'flush all pending file output here

function endFile()

  endFile = ""

end function

Example JScript

//This example script keeps a line counter that

//does not reset for each file

//Our line counter

var a;

//Called for every line in the file

//EOL contains the end of line characters (Unix, DOS or Mac) that must be

//appended to each line

function processLine(line, EOL)

{

a = a + 1;

//add the count to the end of each line

return line + " " + a + EOL;

}

//Called at the start of a processing job -

//perform one-time initialisation here

function startJob()

{

a = 0;

}

//Called at the end of a processing job -

//destroy any declared objects here

function endJob()

{

; //do nothing

}

//Called before each file is opened -

//perform per-file initialisation here

function startFile()

{

return "";

}

//Called before each file is closed -

//flush all pending file output here

function endFile()

{

return "";

}


Scripting automation object

 Scripting automation object
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The TextPipe automation object ('TextPipe') can be called from a VBScript or JScript script to request information from TextPipe or to control processes and log information. It is only available within TextPipe. To make use of this control with Windows Vista you may have to disable User Access Control (UAC).

See also

Automating TextPipe from another program

Normal Functions

property Application : TextPipe.Application

Returns the Application object

property Parent : TextPipe.FilterWindow
property FilterWindow : TextPipe.FilterWindow

Returns the FilterWindow object, from which the output files and other details can be extracted

sub logError( messageText : string )

Logs an error message string to the log file and to the Status Window e.g.

function processLine(line, EOL)

processLine = line & EOL

TextPipe.logError( line & " found " )

end function

sub logWarning( messageText : string )

Logs a warning message string to the log file e.g.

function processLine(line, EOL)

processLine = line & EOL

TextPipe.logWarning( line & " found " )

end function

sub logInfo( messageText : string )

Logs an information message string to the log file e.g.

function processLine(line, EOL)

processLine = line & EOL

TextPipe.logInfo( line & " found " )

end function

function inputFilename() : string

Returns the name (without the drive or path) of the file being processed e.g.

function processLine(line, EOL)

processLine = TextPipe.inputFilename & " - " & line & EOL

end function

function fullInputFilename() : string

Returns the name (with the drive and path) of the file being processed.

function processLine(line, EOL)

processLine = TextPipe.fullInputFilename & " - " & line & EOL

end function

function fullOutputFilename() : string

Returns the name (with the drive and path) of the file where output will go. Note: The file may not actually exist yet.

function version() : string

Returns the TextPipe version number e.g.

function startFile()

dim crlf

crlf = chr(13) & chr(10)

startFile = _

   "TextPipe version " & TextPipe.version & crlf & _

"Input filename=" & TextPipe.inputFilename & crlf & _

"Full input filename=" & TextPipe.fullInputFilename & crlf & _

crlf

end function

sub terminate()

Terminates the current filter thread.

function processLine(line, EOL)

if EOL = "" then

TextPipe.logError( "Line with no EOL found " )

   TextPipe.terminate

else

   processLine = line & EOL

end if

end function

function getGlobalVar( name : string ) : string

Returns the string value of a global variable. Name is not case-sensitive, and multi-line values can be stored.

Global variables are shared between all filters in the same thread e.g.

function processLine(line, EOL)

processLine = TextPipe.getGlobalVar( "myvariable" ) & line & EOL

end function

sub setGlobalVar( name, value : string )

Sets the value of a global variable called "name" to value. Name is not case-sensitive, and multi-line values can be stored.

Global variables are shared between all filters in the same thread e.g.

function processLine(line, EOL)

TextPipe.setGlobalVar "myvariable", line & EOL

'still output the text unaffected

processLine = line & EOL

end function

sub terminateTextPipe()

Terminates TextPipe. Warning! This function may have undesirable side-affects.

sub finishedWithFile()

Halts any further reading of the current input file, and moves to the next file. This is used when you only wish to process the first part of a file, and ignore the remainder, such as processing the first 10 lines of a 200 MB file.

Note: This function does not terminate processing immediately. The current disk block is still processed, but no more blocks are read from disk.

sub alert( message : string )

Displays a message in a window, with an OK button to close it. This helps JScript users debug their code - as there is currently no JScript alert function as there is for VBScript.

property line : integer

Returns the current line of the text being processed. If the text is inside a restriction, this reports the current line number of the restricted text.

property offset : integer

Returns the current file offset of the text being processed. If the text is inside a restriction, this reports the offset of the restricted text.

property filterFilename : string

Returns the name of the current filter (if it has been saved)

Sub filter functions

The following functions only operate when the ActiveX script filter has sub filters.

function subFilterEntireText( inputText : string ) : string

Applies the sub filters to inputText as though inputText was an entire file in its own right. If you need more control over the sub filters use the next set of functions e.g.

function processLine(line, EOL)

lineNumber = lineNumber + 1

if (lineNumber mod 2) = 0 then

   'an even-numbered line

   processLine = TextPipe.subFilterEntireText(line) & EOL

else

   'an odd-numbered line

   processLine = line & EOL

end if

end function

sub initSubFilters()

Prepares sub filters for processing text. MUST be called before sending data into sub filters.

dim lineNumber

function processLine(line, EOL)

lineNumber = lineNumber + 1

if (lineNumber = 100) then

   'start the special processing on line 100

   TextPipe.initSubFilters

   TextPipe.subFilterChunk( line & EOL )

   processLine = ""

else if (lineNumber > 100) and (lineNumber < 200) then

   'special processing for lines 100..200

   TextPipe.subFilterChunk( line & EOL )

   processLine = ""

else if (lineNumber = 200) then

   'end special processing on line 200

   TextPipe.flushSubfilters

   processLine = line & EOL

else

   processLine = line & EOL

end if

end function

function endFile()

if (lineNumber = 100) then

   endFile = ""

else if (lineNumber > 100) and (lineNumber < 200) then

   TextPipe.flushSubfilters

   endFile = ""

else if (lineNumber = 200) then

   TextPipe.flushSubfilters

   endFile = ""

else

   endFile = ""

end if

end function

function startJob()

  lineNumber = 0

end function

sub flushSubfilters()

Flushes all cached data from all sub filters prior to resuming normal straight-through processing.

sub subFilterChunk( inputString : string )

Sends inputString to the sub filters. Remember to call flushSubfilters at the end of the file or beforehand.


Capture text

 Capture text
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

This filter captures text into a named global variable. Once captured, the variable's value can be output using macros with the filters Add Header/Footer, Add Left/Right Margin, Add Column/Add Bytes. It can also be accessed by the TextPipe.getGlobalVar() function from within the scripting filter.

This filter is commonly used to capture header record data to output on each subsequent detail record line. In general, this filter is placed inside a restriction that controls which text gets captured.

It is VERY important to understand how TextPipe processes data to use this filter successfully. For most applications, this will involve placing this filter inside a Restrict to each line in turn filter. If you do not, the text captured will most likely be the very last value found in the file or in each subfiltered section. If you have any questions about this, please contact us.

Name of global variable

The name of the global variable to save the value to. This name is not case sensitive.

Break on value change

When the value of the global variable is changed, TextPipe displays a window allowing you to review (and edit) the value before continuing.

Reset to initial value

Do not reset - never reset the variable to the initial value unless at the start of the job

Reset at start of each file - set the variable to the Initial Value at the start of each file

Reset at start of each region - set the variable to the Initial Value at the start of each fragment of text coming into a subfilter

Reset for both - set the variable to the Initial Value at the start of each file, and at the start of each fragment of text coming into a subfilter

Initial value

The value to give the global variable at the start of the job.


Comments

 Comments
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special

This filter allows you to put comments into your filter lists. This can be handy for documenting complicated search and replace sequences. Comments are also handy to group filters together that perform a common function. Simply drag and drop the filters onto the comment filter.

The top line is used as the description in the filter list.

The right-click menu on both fields supports Undo, Cut, Copy, Paste, Delete and Select All.


Debug filter

 Debug filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

A debug filter is very handy for debugging filters. When text is passed through this filter, it places the output into a window so that you can see what the text looks like at that stage of the filtering process.

See also

Partial Trial Runs


XSLT Filter

 XSLT Filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

This filter applies an XSLT transformation to XML text. The filter depends on the installed version of MSXML in the Windows OS, http://msdn.microsoft.com/en-us/library/ms759204(VS.85).aspx

If an error occurs, TextPipe records the error code and reason in the error log.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Below is a sample XSLT script, to convert XML data to SQL INSERTscript format.

<?xml version="1.0"?>

<!--

XSL Stylesheet

Transforms NewMMARS feedback files into SQL

-->

<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">

<xsl:output method="text"/>

<xsl:template match="//AMS_DOC_XML_EXPORT_FILE">

<xsl:text>update mmars_return_file set type = 'VCC' where mmars_return_file_id = IDENT_CURRENT('mmars_return_file');&#13;&#10;</xsl:text>

<xsl:apply-templates select='AMS_DOCUMENT'/>

</xsl:template>

<xsl:template match="/AMS_DOC_XML_EXPORT_FILE/AMS_DOCUMENT">

<xsl:if test="VC_DOC_VCUST/DOC_DEPT_CD='TRE' and VC_DOC_VCUST/DOC_UNIT_CD='1402'">

<xsl:text>insert into mmars_vcc (mmars_return_file_id, doc_id, vend_cust_cd, ad_id) values (IDENT_CURRENT('mmars_return_file'),'</xsl:text>

<xsl:value-of select="VC_DOC_VCUST/DOC_ID"/><xsl:text>', '</xsl:text>

<xsl:value-of select="VC_DOC_VCUST/VEND_CUST_CD"/><xsl:text>', '</xsl:text>

<xsl:value-of select="VC_DOC_AD[AD_TYP='PA']/AD_ID"/><xsl:text>');&#13;&#10;</xsl:text>

</xsl:if>

</xsl:template>

<xsl:template match="//AMS_IC_STATUS">

<xsl:text>update mmars_return_file set type = 'VSS' where mmars_return_file_id = IDENT_CURRENT('mmars_return_file');&#13;&#10;</xsl:text>

<xsl:apply-templates select='AMS_DOCUMENT'/>

</xsl:template>

<xsl:template match="AMS_DOCUMENT">

<xsl:if test="DOC_DEPT_CD='TRE' and DOC_UNIT_CD='1402'">

<xsl:text>insert into mmars_vss mmars_return_file_id, batch_id, doc_id, doc_cd, doc_vers_no, doc_func_cd, doc_phase_cd, doc_sta_cd) values (CURRENT_IDENT('mmars_return_file'), '</xsl:text>

<xsl:value-of select="@BATCH_ID"/><xsl:text>', '</xsl:text>

<xsl:value-of select="DOC_ID"/><xsl:text>', '</xsl:text>

<xsl:value-of select="DOC_CD"/><xsl:text>', '</xsl:text>

<xsl:value-of select="DOC_VERS_NO"/><xsl:text>', '</xsl:text>

<xsl:value-of select="DOC_FUNC_CD"/><xsl:text>', '</xsl:text>

<xsl:value-of select="DOC_PHASE_CD"/><xsl:text>', '</xsl:text>

<xsl:value-of select="DOC_STA_CD"/><xsl:text>');&#13;&#10;</xsl:text>

</xsl:if>

</xsl:template>

</xsl:stylesheet>

See also

Database filter


File Shred

 File Shred
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The file shred filter completely overwrites a file, making the original data unrecoverable. Note that there is no point adding more than one File Shred filter to a filter list, as only the data changes from the last filter get written to disk. Instead, run the filter multiple times; as each time the file is overwritten with a different pattern.


Switch CPU

 Switch CPU
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Server Edition Only)

This filter causes a logical break in the processing from one CPU to another. It allows processing load to be distributed across multiple CPUs.

For example, if a filter list involved data preparation, sorting, and then output generation, then it makes sense to have the sorting on one CPU alone, as sorting is very CPU intensive.

Start by placing the splitter roughly in the middle of the filter list – before or after a sort typically. Then try moving it up and down to gauge performance.

Note: The numeric field of this filter is currently unused.

Each time you use this filter, a new logical processing thread is created. Windows takes care of distributing this thread to the CPU with the least load.


Split files

 Split files
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

This filter breaks files at specific boundaries. It is very useful for chopping up large files so that they can be transferred on floppy disk, then joined together using the merge function. Files are numbered with an extension of 000, 001, 002, etc, which can also be controlled.

If the output filter is set to Output to Clipboard or Single File Output, the split still occurs but the results are merged back together by the output filter as the very last stage in the filtering process.

If you end up with unwanted zero-byte files, use the Output Filters's Remove Empty Output Files option.

Split at Size of (bytes)

When selected, files are split when they reach the given size (in bytes). The maximum file size is 2147483647 (2 GB).

Common Sizes

Common disk sizes are provided - choose one from the list and it will be defaulted to the field on the left

Split at Pattern

When selected, files are split when a given pattern is found.

Pattern

The case sensitive perl pattern to split on. If named subexpressions are created by this pattern, they are available as macros to be used in the filename, and are also available to other filters. e.g.
\\id (?P<Book>[^\r\n]+?)

would capture the text following '\id bookname' and place it in a macro called @BOOK@ for use in other filters.

After this many

Split after this many occurrences of the pattern have been found. The maximum is 2147483647.

e.g. if 'After this many' is set to 2, splits will occur after the 2, 4, 6, 8, 10.. matches.

Split Position

Where to split:

· Before pattern - split pattern is output at start of next file
· After pattern - split pattern is output at end of first file
· At pattern - split pattern is removed

Split After Lines

The Split occurs after the specified number of lines have occurred. This is handy for dividing print files up into separate files, with 66 lines per page. The maximum value is 2147483647.

Split Filename

If you wish, you can control the names of the output files. All names are based on the input filename, in the input filename's folder. If you wish to control the output path, you can specify an absolute path here.

Macro name | Use
%f | The input file's filename e.g. 'readme'
%e | The input file's extension without any period e.g. 'txt'
%d | The split file counter, starting from 0.
%3.3d | The split file counter, starting from 0, formatted in a field 3 wide with leading zeros.
%5.5d | The split file counter, starting from 0, formatted in a field 5 wide with leading zeros.
%x | The split file counter in hexadecimal.
@macro | You can insert macros by preceding their name with @

First Filename Number

The number for the first split file.

Prevent Directories from Overloading

Windows get very slow and unstable when there are too many files in one directory. When this option is enabled, TextPipe will start a new folder if the number of split files exceeds 1,000.

New folder names are 1000, 2000, 3000 etc.

See also

Merge


T-Filter (Copy text and send to subfilters)

 T-Filter (Copy text and send to subfilters)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The T-Filter allows you to process the same input in a number of ways.

Let's say you wanted to first count the number of blank lines in your text and save this value to a new file/clipboard or capture variable, and then sort the text removing duplicate lines.

You would add a T-Filter, and as a subfilter of it, you would place the filters to identify blank lines (e.g. a restrict to matching lines with pattern text of '^$'), a count lines filter and the appropriate output filters such as a Secondary Output filter.

Once this side of the T has finished processing, the data is discarded, and the original text continues processing just as though the T-Filter did not exist.

[****]


Merge (join) files

 Merge (join) files
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The merge files filter is used to join files together. The content from each file is simply appended to the content from the previous file, with no extra characters e.g. if you joined files A, B and C, the merge files filter would result in output like this:

[file A][file B][file C]

This filter can be used for:

· Joining together binary files that have been split
· Joining together the processed results from large numbers of smaller files e.g. extracting email addresses or URLs from a large number of web pages, then using the merge files filter to join the results together so that a sort and remove duplicates can be performed.

By contrast, the output filter joins all input files together when it is set to Output to Clipboard or Single File Output. The join is performed as the very last stage in the filtering process. A merge files filter allows the merge operation to be followed by additional text processing operations, typically a sort and remove duplicates, or adding a header or footer. If the output filter is also set to Output to Clipboard or Single File Output, the merge filter filename is overridden by the output filter. The content from all files is joined at the point that the merge filter appears in the filter list.

If you join text files together, you can ensure that each file ends properly (i.e. that the start of subsequent files are not joined to the end of a line of the previous file) by using an Add Footer filter (with \013\010) before the Merge files filter.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.

See also

Split

Output Filter


Seconday output filter
 Seconday output filter
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

A Secondary Output Filter behaves just like an Output Filter, but can be used anywhere in the filter list.

It's purpose is to take a restricted set of output (e.g matching lines, non-matching lines etc) and send it to a new file.

It is also commonly used to match records from the Mainframe Copybook Wizard and send them to a new file.

Commonly, this filter is followed by a Remove All filter to prevent the output re-entering the text stream.


Count duplicate lines

 Count duplicate lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The Count Duplicate Lines filter outputs a count of the number of times a line has been repeated (only if there are 2 or more occurrences). The file need NOT be sorted prior to this filter. A single copy of each duplicated line is output, with a count of the number of duplicates at the start of each line. From a set of duplicate lines, the line that gets output is the first duplicate line of the set.

Ignore case

If ignore case is checked, lines do not need to be cased identically to be considered duplicates. Two identical lines, one in upper case, and one in lower case, would be considered duplicates and counted together by this filter. If ignore case is unchecked, the lines must be absolutely identical to be considered duplicates. The case checking routines are ANSI aware, so their behaviour may change depending on your locale.

Start column

The comparison can also ignore leading characters if desired by setting the start column higher than 1. This can be used to skip line numbers, which can be used to find duplicates that are not adjacent. To skip line numbers, set the Start Column to 6 (or so), and set the length to 4096, or a length greater than your maximum line length.

Length

The comparison can also ignore trailing characters if desired, by setting the length to less than the length of the line.

Include counts of 1

Normally this filter only outputs lines with counts of 2 or more (ie, they are duplicates). When this box is checked, counts of 1 are included as well.

Output Format

Here you specify how the results are output.

%d is the count, %s is the line it refers to. So

%d %s | The count, a space, the string
%d\t%s | The count, a tab, the string
%1:s\t%0:d | The string (normally second), a tab, the count (normally first)
%d,%s | The count, a comma, the string

See also

Remove duplicate lines

Remove lines found in file

Retain lines found in file


Move columns

 Move columns
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

TextPipe will move columns to a new position on the line. The new position is specified assuming that the moved columns have been removed from the line.

Move columns

See the Remove columns filter for how to specify which columns to move.

Move to

The position to insert the moved columns. The new position is specified assuming that the moved columns have been removed from the line.

See also

Special Menu

Move columns

Move fields (CSV, Tab, Pipe etc)


Move delimited fields (CSV, Tab, Pipe etc)

 Move delimited fields (CSV, Tab, Pipe etc)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

TextPipe will move CSV-delimited fields to a new position on the line. The new position is specified assuming that the moved fields have been removed from the line. TextPipe ensures that all the delimiters on the line are correctly maintained, both at the end of the line and where the moved fields are inserted.

Move fields

See the Remove delimited fields (CSV, Tab, Pipe etc) for how to specify which fields to move.

Move to

The position to insert the moved fields. The new position is specified assuming that the moved fields have been removed from the line.

See also

Special Menu

Move columns


Copy columns

 Copy columns
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

TextPipe will copy columns to a new position on the line.

Copy columns

See the Remove columns filter for how to specify which columns to copy.

Copy to

The position to insert the copied columns.

See also

Special Menu

Copy columns

Copy fields (CSV, Tab, Pipe etc)


Copy delimited fields (CSV, Tab, Pipe etc)

 Copy delimited fields (CSV, Tab, Pipe etc)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

TextPipe will copy CSV-delimited fields to a new position on the line. TextPipe ensures that all the delimiters on the line are correctly maintained, both at the end of the line and where the copied fields are inserted.

Copy fields

See the Remove delimited fields (CSV, Tab, Pipe etc) for how to specify which fields to copy.

Copy to

The position to insert the copied fields.

See also

Special Menu

Copy columns

Copy fields


Repeat Subfilters X Times

 Repeat Subfilters X Times
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The subfilters are repeated the specified number of times. Useful when you are not sure how many times a search/replace pattern needs to be applied.

See also

Reverse line order


Repeat file

 Repeat file
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The file's contents are repeated the specified number of times. This filter needs to keep a copy of the entire file in memory so it can output it a given number of times. This filter is useful for generating test data sets of a given size.

See also

Reverse line order


Randomize lines

 Randomize lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

This filter put lines into random order. This is useful when a random sample of data is required for statistical purposes - just follow this filter with a head/tail of file filter.

The lines output will differ from one run to the next. The order the lines are chosen is determined by a pseudo-random number generator.


Reverse each line

 Reverse each line
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

Each line is output reversed from left to right. This can be useful to

· extract domain names from web site log files - use this filter to reverse each line, use an extract matches filter of [\w\d]+\.[\w\d]+ to extract each domain name, then reverse each line again.

Note: This filter will NOT work on Unicode or UTF-8 data. It will only work on single-byte data such as ASCII or ANSI.

See also

Reverse line order


Reverse line order

 Reverse line order
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Special (Std and Pro version only)

The order of the input lines is reversed i.e. the last line comes out first and the first line comes out last.

A file is read entirely into RAM before being reversed, so be wary of reversing files that are larger than your machine's RAM size.

See also

Reverse each line


Map Filters

 Map Filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps

These filters map each character of input to an output character or string.

A map defines a mapping for every single input character (all 256 of them). Each ASCII code is represented by a row in a grid – the left column gives the ASCII code in decimal ("Dec"), the next column gives the ASCII code in Hex ("Hex"), the next column shows the ASCII representation of the character ("ASCII"), and the right hand column shows the string that it will be mapped to ("Replacement text"). Note that if the system font does not provide a definition for a particular ASCII character (as is usually the case for ASCII codes 0 to 31 and 127, but there may be more), then it is represented as a thick vertical bar. Cells that are output differently to the input are shown with a yellow background - this makes it easy to scan through the list looking for changes.

When the replacement text cannot be normally displayed, characters are displayed using an escape code e.g. \r\n is the code for a DOS end of line. Any combination of characters can be entered in this way. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy and entering special characters.

If all the replacements are a single byte, then TextPipe will used an optimized filter which is 3 times faster than the custom case (which handles replacements of zero characters, one character or more).

Reset

A cell is Reset when it passes through unchanged i.e. the output character is the same as the input character.

Reset All

All cells can be reset by clicking "Clear All".

Clear

A cell is cleared or empty when nothing is output when the character is encountered in the input stream.

Clear All

All cells can be cleared by clicking "Clear All".

Reverse

If the character mapping is 1:1 (i.e. each input character gets mapped to a single output character), the "Reverse" button can be clicked to form the Reverse mapping translation.

Scroll to

Clicking in the Scroll To field and pressing a keyboard character will cause the grid to immediately scroll to show the mapping for that character. This makes it easy to find characters when you're not sure where in the list they appear.

Open and Save

Maps can be saved to and opened from .MAP files by clicking on the Save and Open icons displayed when a map filter is selected for editing.

See also

Refresh map list

New map

Base map on filter list

Available maps

Unicode maps


New map

 New map
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps

Adds a new map filter, using values from the last edited map as defaults.

See also

Refresh map list

Base map on filter list

Available maps

Unicode maps


Unicode maps

 Unicode maps
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps (Standard and Pro)

A Unicode map allows 2 byte (UTF-16) or 4 byte (UTF-32) Unicode characters to be mapped to a sequence of zero or more output characters in the same way that single byte characters are mapped. The primary difference between ANSI maps and Unicode maps is that a Unicode map only specifies characters that are changed - non-specified characters are controlled by a separate setting (otherwise you would have an unwieldy list of 65536 to 1 million entries to edit and work with).

The Unicode map expects characters to be found in big endian order, i.e. MSB first, so the Unicode code point x00FA would be found in the file as two consecutive characters, 0x00 0xFA, NOT 0xFA 0x00. If the Unicode map does not work for you, you can check your input file format using the Convert\Hex Dump filter, and if necessary, use the Swap UTF-16 word order or Swap UTF-32 word order filter.

A Unicode map filter looks like this:

Type a value in the output string column to define how a Unicode character gets remapped. Click the up and down buttons in the Start Range (Hex) or End Range (Hex) columns or type a new number to change the range of Unicode characters being remapped. New values can be added on the last row, or using the Populate Values panel.

Delete Selected Rows

Deletes the currently selected rows from the grid. Character ranges not defined in the grid are controlled by the Non-entered characters panel.

Open Map File

Opens a Unicode map file, replacing the existing map grid (not the non-entered characters panel values). Maps can be loaded from Excel worksheets (.XLS), Comma Separated Value files (.CSV, the default when the file extension is not recognized) and Tab delimited value files (.TAB).

Save Map to File

Saves the current Unicode map grid to a file (not the non-entered characters panel values). Maps can be saved to Excel worksheets (.XLS), .CSV Comma Separated Value file (the default) or a .TAB Tab-delimited value file (only when the file extension is .TAB). Values are saved exactly as they are shown on screen.

UTF Mode

This drop down specifies whether each input character is 2 bytes (UTF-16) or 4 bytes (UTF-32).

Non-entered characters

This panel controls what happens to characters not found in any range in the grid.

· Pass through unchanged - characters not found in the grid are passed through unchanged
· Pass to sub filter. Typically the sub filter is a Script filter - see below for a sample script.
· Replace with value - characters not found in the grid are replaced with the set value. This fields supports special characters. Using %x (or %4.4x for better formatting) in the replacement value as a placeholder for the hex value of the character. Using %d in the replacement value is a placeholder for the decimal value of the character. e.g.

Invalid character found 0x%4.4x

It's also common to set the Replace with value to

&#%d;

to convert it to a Unicode entity expressed in decimal, or

&%4.4x

to a Unicode entity expressed in hex, or

\U+%4.4x

to a Unicode character value.

· Remove - characters not found in the grid are removed

Populate values

The populate values group makes it easy to set large groups of values. First enter in a starting value and an ending value (in decimal), then click the button corresponding to your choice:

[Clear] Add the specified characters to the grid and set their mapping so that nothing is output when they are encountered.

[Default] Add the specified characters to the grid, and set their mapping so that they are passed through unchanged.

Sample sub filter VBScript

'Log unknown Unicode values to TextPipe error log

dim a

dim errorCount

const maxErrors = 20

function processLine(line, EOL)

if len(line & EOL) = 2 then

'line & EOL is the two characters of the Unicode code point

errorCount = errorCount + 1

if errorCount < maxErrors then

a = Asc(mid(line & EOL,1,1)) * 256 + Asc(mid(line & EOL,2,1))

TextPipe.logError "Invalid code point 0x" & hex( a )

'msgBox "Invalid code point 0x" & hex( a )

elseif errorCount = maxErrors then

TextPipe.logError "Maximum code point Errors reached: " & errorCount

end if

end if

'return nothing - invalid code point is absorbed

processLine = ""

end function

sub startJob()

errorCount = 0

end sub

sub endJob()

end sub

function startFile()

startFile = ""

end function

function endFile()

endFile = ""

end function

See also

Refresh map list

New map

Base map on filter list

Multi-maps

Available maps

Swap UTF-16 word order

Swap UTF-32 word order

Make Big Endian

Make Little Endian

Unicode conversion


Multi-maps

 Multi-maps
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps

A multi-map is a specialized version of a Unicode Map. It is used in font remapping applications to take a single input file, and map it to one or more output font sets. The hierarchy of the mapping can be controlled at run-time from the command line.

As each UTF-16 or UTF-32 character is processed, TextPipe first checks if it is marked as INVARIANT. If so, TextPipe scans languages in MULTIMAPORDER sequence for the first language with a non-blank entry, and outputs it without a language change sequence. If the character is not INVARIANT, TextPipe scans languages in MULTIMAPORDER sequence for the first language with a non-blank entry, outputting a language change sequence if required, the found value, and then resets the current language.

The Excel spreadsheet has individual UTF-16 Unicode hex code points in the left column (e.g. 0021, 003c), and a series of values in named columns or language families to the right (e.g. 01, 08, 09), starting at column 3. By setting the order that the languages are used (using MULTIMAPORDER), the same Excel spreadsheet can be used to map to different output languages.

Here is an example Excel multi-map:

UTF-16 | Invariant | 01 | 08 | 09 | 10 | 11 | 12 | 32 | 66
0009 | ON | \x09 | \x09 | \x09 | \x09 | \x09 | \x09 | \x09 | \x09
000A | ON | \x0A | \x0A | \x0A | \x0A | \x0A | \x0A | \x0A | \x0A
000D | ON | \x0D | \x0D | \x0D | \x0D | \x0D | \x0D | \x0D | \x0D
0020 | ON | \x20 | \x20 | \x20 | \x20 | \x20 | \x20 | \x20 | \x20
0021 |  | \x21 | \x21 | \x21 | \x21 | \x21 | \x21 | \x21 | \x21
003C |  | \x3C | \x3C | \x3C | \x3C |  |  | \x3C | \x3C
003D |  | \x3D | \x3D | \x3D | \x3D | \x3D |  | \x3D | \x3D
003E |  | \x3E | \x3E | \x3E | \x3E |  |  | \x3E | \x3E
003F |  | \x3F | \x3F | \x3F | \x3F | \x3F | \x3F | \x3F | \x3F
0040 |  | \x40 | \x40 | \x40 | \x40 | \x40 |  | \x40 | \x40
0041 |  | \x41 | \x41 | \x41 | \x41 | \x41 | \x41 |  |

The multi-map order command line options specifies the precedence order given to the columns. The first column specified is called the primary. If only the primary is specified, then the remaining columns are used in left-to-right order (ignoring the primary column). If two or more columns are specified, then only those columns are searched for values.

See also

Refresh map list

New map

Base map on filter list

Available maps


Base map on filter list

 Base map on filter list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps

Adds a new map based on the filters currently in the filter list. This is very useful if you need to customise any of the existing filters eg the EBCDIC -> ASCII filter. By basing a new map on the current filter list, you can effectively "copy" the existing filter, and then tweak it as desired.

See also

Refresh map list

New map

Available maps

Unicode maps


Refresh map list

 Refresh map list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps

If you copy or move map files to TextPipe’s folder, use this item or press F5 to force TextPipe to display it.

See also

New map

Base map on filter list

Available maps

Unicode maps


Available maps

 Available maps
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Maps

Any maps that have been defined by the user (and are saved in the same folder as TextPipe.exe) will be listed in this menu.

To add a saved map to the filter list, just select it from this menu.

See also

Refresh map list

New map

Base map on filter list

Unicode maps


Restriction filters

 Restriction filters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict

These filters restrict the amount of text that is affected by sub filters i.e. without affecting the main text.

Restrict to line range

Restrict to column range

Restrict to matching lines

Restrict to non-matching lines

Restrict to delimited fields (CSV, Tab, Pipe etc)

Restrict to HTML/XML element or attribute

Restrict to lines from list/NOT from list

Restrict to lines matching/not matching pattern list

Restrict to inside filename

Restrict to filenames matching pattern/not matching pattern

Restrict to each line in turn

Restrict to blocks

Restrict to byte range

Restrict to ANSI/Unicode type files (Pro version only)

Note - sub filters treat each section of restricted text as a complete file.

See also

Lesson 5 - Restrictions and sub filters


Restrict to line range

 Restrict to line range
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict

This filter restricts sub filters to operate on a range of lines. The lines can be selected by using an offset from the beginning or end of the file. All selected lines can be completely removed by using the Delete All filter as the sub filter. This filter can be used like a Unix head or tail filter.

See also

Sub filters

Restriction filters


Restrict to column range

 Restrict to column range
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

This filter restricts sub filters to operate on a range of columns. The columns can be selected by using an offset from the beginning or end of the line. All selected columns can be completely removed by using the Delete All filter as the sub filter. This filter can be made a sub filter of a Restrict line range filter.

If you're working with EBCDIC data, you should not use this filter, because EBCDIC data can contain embedded new line characters in packed or zoned decimal fields. Use the Restrict to Byte range filter instead.

See also

Sub filters

Restriction filters


Restrict to matching lines

 Restrict to matching lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter restricts sub filters to operate on lines that match a pattern or regular expression.

Note: This filter allows you to use the regular expression characters, ^ and $.

^ match from the start of a line (or at the start of the file)

$ match at the end of a line (or at the end of the file)

Note 2 - this filter will not match blank lines.

Restrict to Lines Matching

The string or pattern to find. The string can be found anywhere on a line. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

Pattern type

Determines which style of pattern match to use. The default is to use powerful perl-style pattern matching, but you can also use newer and easier EasyPatterns, as well as Word-style and Brief-style patterns. Older egrep-style pattern matching is offered for backward compatibility.

Match Case

When checked, the case of the string found must be identical to the Match Pattern exactly.

Output Count of Matches

When checked, outputs a count of the number of matches in each file instead of the actual matches.

Include Filename

When checked, outputs the filename prior to showing the match information.

Include Line Numbers

When checked, shows the line numbers of each match with the match text.

UTF-8 Support

Allows the pattern and the text being searched to be in Unicode UTF-8 format.

Ignore Empty Matches

Ignores any matches that do not contain any text e.g. EasyPattern [ 0 or more characters ] can contain zero characters.

Context Lines

Context Lines Before the Match

This option outputs context lines prior to the matching line. This can be very useful when you want to see the lines surrounding a matching line. Enter a value larger than 0 to determine the number of context lines to be output. TextPipe may not be able to output the desired number of context lines before the matching line if the matching line is near the start of the file. In this case TextPipe will output as many prior context lines as are available. Note also that if two subsequent lines match then the resulting output may be difficult to understand.

Context Lines After the Match

This option outputs context lines after the matching line. This can be very useful when you want to see the lines surrounding a matching line. Enter a value larger than 0 to determine the number of context lines to be output. TextPipe may not be able to output the desired number of context lines after the matching line if the matching line is near the end of the file. In this case TextPipe will output as many trailing context lines as are available. Note also that if two subsequent lines match then the resulting output may be difficult to understand.

See also

Extract matches

Restrict to non-matching lines

Restriction filters


Restrict to non-matching lines

 Restrict to non-matching lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter restricts sub filters to operate on lines that DO NOT match a pattern or regular expression.

This filter is the exact opposite of the Restrict to matching lines filter. The right-click (or context) menu on this field supports Undo, Cut, Copy, Paste, Delete, Clear Entire Field, Select All and Select All And Copy, entering special characters and inserting pattern match characters.

See also

Extract matches

EasyPattern matching

Perl pattern matching

Restrict to matching lines

Restriction filters


Restrict to delimited fields (CSV, Tab, Pipe etc)

 Restrict to delimited fields (CSV, Tab, Pipe etc)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

This filter restricts sub filters to operate on a range of fields, delimited by a given character. Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data.

Choose the delimiter that separates your fields

You can choose a predefined delimiter character, or select your own. The trailing delimiter (if any) is included in the restricted text, so the entire field can be completely removed by using the Delete All filter as the sub filter (or you can use a Remove delimited fields (CSV, Tab, Pipe etc) filter). When Comma (.csv) is chosen, TextPipe automatically handles single and double quoted strings, with embedded line feeds.

First Row Contains Field Names

If the first line of the file contains Field Names, check this box so that TextPipe can count how many fields are expected. It can then determine if a field has embedded CR/LF characters and spans multiple lines. TextPipe can also determine this without a header if the fields are properly double-quoted - TextPipe will notice the missing double quote and continue reading the record from the following line.

Restrict Fields

Several different means of specifying the fields to restrict are provided:

1. Start field, end field. This removes all text including and between the specified fields.
2. Start field, Length. Removes Length fields starting from (and including) field Start.
3. End field, Length. Removes Length fields backwards starting from (and including) field End.
4. Start field to end of line. Removes all fields from the Start field to the very end of the line.
5. Length to end of line. Removes Length fields backwards starting from (and including) the last field.

Process fields individually

When this field is unchecked, the entire field range is returned as a single string. It is often easier for the sub filters to process each field individually, in which case it is checked, and the next two options apply.

Exclude column delimiter

When checked, this excludes any trailing delimiter from the restricted text, making it easier for sub filters to work with.

Exclude quotes (if present)

When checked, this excludes any surrounding quotes from the restricted text, making it easier for sub filters to work with.

See also

Sub filters

Restriction filters

[****]


Restrict to lines from list

 Restrict to lines from list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter restricts sub filters to operate only on lines that are in/NOT in the given list. The comparison is NOT case sensitive.

The sub filters can be used to insert a marker code such as ##DELETE## or ##KEEP## which can be used by a subsequent Retain matching lines or Remove matching lines filter.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.


Restrict to lines matching pattern list

 Restrict to lines matching pattern list
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter restricts sub filters to operate only on lines that match/do NOT match the given list of regular expressions.

The sub filters can be used to insert a marker code such as ##DELETE## or ##KEEP## which can be used by a subsequent Retain matching lines or Remove matching lines filter.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.


Restrict to inside filename

 Restrict to inside filename
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter uses sub filters to operate on the filename (drive + path + filename) itself. Using this filter, you can rename files, change their folders, UPPERCASE or lowercase the filenames, search and replace parts of the filename and much more!

The output filename is set before the file contents are processed, so while it is possible to uppercase or search/replace inside the filename, it's not possible to import an external value into it. The reason for this is that the output file so far (ie temp file) MUST be placed in the correct folder (and/or on the correct output drive/device) rather than the system having to move or copy it afterwards with ensuing problems of access rights and disk space.

Note - because sub filters treat each section of restricted text as a complete file, you cannot use a normal line number filter to add numbers to filenames. Instead, use a VBScript-based line number script.

If you are trying to generate a filename list, use a filter of this form:

|--Remove all

|

|--Add header [@fullInputFilename\r\n]

|

|--Merge output to file c:\mergefilename.txt

If you only wish to extract a list containing certain files, either change the file specification in the files grid, or use the filter list below. Important! Ensure you include the final Remove All filter, which prevents TextPipe reading the entire file.

|--Restrict to file name

| |

| +--EasyPattern [.[ 'gif' or 'jpg' ]] with [$0\r\n]

| | [ ] Match case

| | [ ] Whole words only

| | [ ] Case sensitive replace

| | [ ] Prompt on replace

| | [ ] Skip prompt if identical

| | [ ] First only

| | [ ] Extract matches

| | Maximum text buffer size 4096

| |

| +--Merge output to file c:\filenames.txt

|

|--Remove all

|

|--Output to clipboard

See also

Sub filters

Restrict to delimited fields

Restriction filters


Restrict to filenames matching pattern/not matching pattern

 Restrict to filenames matching pattern/not matching pattern
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Pro and Std only)

This filter applies its subfilters only to files with filenames (ie drive + path + filename) matching or not matching a pattern or list of patterns. This is very handy for only applying a Convert Word Documents to Text filter only to files matching the pattern

\.DOC$

With the appropriate pattern, this filter can also be used to control subfilters based on filename, folder and drive.

Note that this filter uses case-insensitive perl patterns, not DOS wildcards.

See also

Sub filters

Restrict to delimited fields

Restriction filters


Restrict to zip internal filenames matching pattern

 Restrict to zip internal filenames matching pattern
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Pro and Std only)

This filter applies its subfilters only to files inside zip files, with filenames (ie path + filename) matching a pattern or list of patterns. This is very handy for only applying a XML replacements to the files inside Office 2007+ documents.

Note that this filter uses case-insensitive perl patterns, not DOS wildcards.

See also

Sub filters

Restrict to delimited fields

Restriction filters


Restrict to each line in turn

 Restrict to each line in turn
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter restricts sub filters to operate on each line in turn. This filter is used for its side effect of limiting the matched text to a single line at most.

See also

Sub filters

Restriction filters


Restrict to blocks

 Restrict to blocks
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

A Restrict to Blocks filter groups every N bytes together and sends each group to its sub filters. Each group appears to be a "new file" to the sub filters.

For example, a Restrict to Bytes filter specifies a range that only occurs once in each file e.g. Bytes 1 to 3. By placing the Restrict to Bytes filter inside the Restrict to Blocks filter, the Byte range effectively occurs multiple times -

1. Bytes 1 to 3 of the first group
2. Bytes 1 to 3 of the second group
3. Bytes 1 to 3 of the third group
4. and so on.

If you have a known record length, use this filter to ensure each record is processed as a single unit.

When combined with a Secondary Output Filter as a sub filter (with Append Mode on), records can be selectively written to a new file.

See also

Sub filters

Restriction filters


Restrict to blocks of lines
 Restrict to blocks of lines
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter groups lines together so they can be acted on by a subfilter. e.g. to insert a footer after every 66 lines.


Restrict to byte range

 Restrict to byte range
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter restricts sub filters to operate on a range of bytes. The bytes can be selected by using an offset from the beginning or end of the containing subfilter, usually a Restrict to blocks filter. Selected bytes can be completely removed by using the Delete All filter as the sub filter.

This filter is handy for limiting the data sent to a packed decimal or zoned decimal expansion filter.

See also

Sub filters

Restriction filters


Restrict to ANSI/Unicode type

 Restrict to ANSI/Unicode type
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Found under Filters\Restrict (Std and Pro version only)

This filter checks the type of a file before applying its subfilters. There are three main classes of file type identified:

· ANSI (note: UTF-8 files marked with a Byte Order Mark are not processed as ANSI files) (Standard and Pro)
· UTF16, both LE and BE. To enforce a consistent Byte Order, use the Make Big Endian or Make Little Endian filters (Standard and Pro)
· UTF32, both LE and BE. To enforce a consistent Byte Order, use the Make Big Endian or Make Little Endian filters (Standard and Pro)

See also

Restriction filters


File Menu

 File Menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

New

Opens a blank new filter window. New filters are created using the normal template, normal.fll, found in TextPipe's installation folder.

Open

Opens a saved filter list file.

To open a sample filter:

1. Click on Open in the File menu
2. TextPipe displays the filter list files in the TextPipe folder, usually My Documents\DataMystic\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
3. Select a filter list file (.fll file) to open, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to open
4. Click the Ok button
5. TextPipe will open and display the selected filter list file.

Open in same window

Opens a filter in the same edit window, without modifying the file list. This can be used if you load a file list first either by drag and drop or from the Window Explorer context menu, and then need to change the filter without modifying the files.

Search

Opens a Windows Explorer search dialog box in TextPipe's installation folder. This can be used to quickly search for filters.

Insert

Inserts a saved filter list at the current cursor position.

Sometimes you need to incorporate the filters that you have saved in a filter list file. To copy them in to the current filter:

1. Position the cursor on top of the filter where you want the inserted filter list to appear. The inserted filter list will appear ABOVE the currently selected filter
2. Click Insert in the File menu
3. TextPipe displays the filter list files in the TextPipe folder, usually My Documents\DataMystic\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
4. Select a filter list file (.fll file) to insert, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to insert
5. Click the Ok button
6. TextPipe will insert the new filter list above the currently selected filter.

Save

Saves the current filter list. If no name has been given to it, a Save As is performed.

Save as

Saves the current filter list with a new name.

Close

Closes the current filter window (prompting to save if any changes have been made).

Close all

Closes all window (prompting to save if any changes have been made).

Link to filter (Pro version only)

Incorporates the filters found in an external filter file at run time. This means than any changes that have been made to the linked filter list will be reflected in any filter list that links to it.

To link to a filter list:

1. Position the cursor on top of the filter where you want the linked filter list to appear. The linked filter list will appear BELOW the currently selected filter
2. Click Link to filter in the All Available Filters view of the Filters to Apply tab (Pro version only)
3. TextPipe displays the filter list files in the TextPipe folder, usually My Documents\DataMystic\TextPipe. Navigate to the TextPipe folder if the folder displayed is not the TextPipe folder
4. Select a filter list file (.fll file) to link to, or navigate into one of the subject area subfolders such as coding, conversions, email, etext, fun, html or sysadmin and select a filter list file to link
5. Click the Ok button
6. TextPipe will insert a new link filter BELOW the currently selected filter.

Notes: When you link to a filter file, its Input and Output Filters are ignored, and all its Files to Process are ignored. Instead, the Input and Output filters are overridden by the Input and Output Filters of the filter list linking to it.

If the linked filter list performs special output functions in the Output Filter, such as merging the result to a new file, then when it is linked this function will no longer work (because the Output Filter is ignored). You cannot use a linked filter to re-process the original files in their unmodified state - whatever the linked filter does (including merging files together) is passed on to the next filter. The best way to achieve this is to run two separate filters consecutively from a batch file.

Export

Export filter to clipboard (Pro version only)

Uses the same output format as printing the filter list, but sends it to the clipboard so that it can be pasted into other programs. This feature is a very handy way of maintaining text-based revision control of your filters. Using a text-based diff program you can compare two filters easily after exporting each to the clipboard.

Export command line (Pro version only)

Generates command line parameters corresponding to the current filter list, and allows them to be saved to a batch file (.BAT). Note that not all filters are supported via the command line. This can be handy when editing complex filters externally to TextPipe. Note that the hierarchical structure of the filter list is NOT maintained.

Export to VBScript (Pro version only)

Generates VBScript code corresponding to the entire current filter list, and allows it to be saved to a VBScript file (.VBS). ALL filters are supported, and the hierarchical structure of the filter list IS maintained. This can be very handy when editing complex filters externally to TextPipe, and for creating code to inserting into Web pages, ASP scripts or other scripting application.

Export to JScript (Pro version only)

Generates JScript code corresponding to the entire current filter list, and allows it to be saved to a JScript file (.JS). ALL filters are supported, and the hierarchical structure of the filter list IS maintained. This can be very handy when editing complex filters externally to TextPipe, and for creating code to inserting into Web pages, ASP scripts or other scripting application.

Export to web page (Pro version only)

Generates a sample web page with both VBScript and JScript code corresponding to the entire current filter list, and allows it to be saved to a web page (.HTM). Sample edit boxes are also provided for testing the filter on text. This can be very handy as a template to create your own TextPipe Lite-based front-end - making it easier for non-technical users.

Sent to...

If Outlook is installed, this attaches the current filter to an email ready to send to DataMystic support.

Print

Prints the current filter list. The output shows the names of the filters and the option settings, and the file list.

Printer setup

Allows the page orientation etc to be configured for the printer.

Recent files

Recent files is a submenu that shows a list of previously opened filter files, if any. A previous filter file can be opened quickly by just clicking on it in the menu. The maximum number of entries shown can be controlled through the preferences dialog.

Exit

Leaves TextPipe. You will be prompted to save the current filter list if you have modified it. Clicking [Yes] saves the filter and exits, clicking [No] does not save the filter and still exits, and clicking on [Cancel] does not exit.


Link to TextPipe filter file (.fll)

 Link to TextPipe filter file (.fll)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This filter incorporates the filters found in an external file. By using the feature, you can build small filters that perform useful sub-functions, and then use the linking feature to embed them in one larger filter. Changes to the subfilter will be reflected in all filters that link to it.

For example, you might have a filter than you always use for formatting paragraphs. You might link to it in other filter that reformat pages, number lines or generate lists. You use the Link Filter capability to generate reusable chunks of filters.

The filename can be specified using a path relative to the location of the current filter (e.g. ..\html\ConvertHTMLEntities.fll), or an absolute path e.g. c:\Program Files\DataMystic\TextPipe\Filters\html\ConvertHTMLEntities.fll

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes. A path can be specified relative to the current filter path.


Edit Menu

 Edit Menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Undo changes to this line

Restores the previous values for filter settings displayed in the right-hand window.

Cut

Cuts selected text to the Windows clipboard. If the current control is the filter tree, this cuts the selected filter and its sub filters to the clipboard.

Copy

Copies selected text to the Windows clipboard. If the current control is the filter tree, this copies the selected filter and its sub filters to the clipboard. To Copy an entire filter tree to the clipboard, right click the top-most node (logging options) and choose Copy.

Paste

Pastes text from the Windows clipboard to the current cursor position. TextPipe escapes special characters e.g. for a single line edit control, carriage return/line feeds are escaped to \r\n. If the current control is the filter tree, and a filter has been Cut or Copied to the clipboard, this pastes the filters into the filter tree.

Paste special

Pastes text from the Windows clipboard without any intervention from TextPipe (normally TextPipe escapes special characters e.g. for a single line edit control, carriage return/line feeds are escaped to \r\n).

Delete

Deletes the currently selected text.

Duplicate current filter (x2)

Copies the filter underneath the cursor (and all of its sub filters) and inserts it below the current cursor position. The [x2] button can also be clicked to perform the same action.

Clear clipboard

Clear the current clipboard contents.

Select All

Selects all the text in the current control. Has no effect if the current control is the filter tree. To Copy an entire filter tree to the clipboard, right click the top-most node (logging options) and choose Copy.

Toggle filter enable/disable

Toggles the filter under the cursor between an enabled and disabled state. When disabled, a filter takes no part in filter processing. Disabled filters are shown with a red cross over their icon. This menu item is only available when the Filters to Apply tab is displayed.

Filters can also be toggled by double clicking on them in the filter list.

Preferences

Displays the preferences dialog


Preferences Dialog

 Preferences Dialog
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Play Sound On Completion

When enabled, the specified sound is played when a TextPipe job completes. This is useful for long jobs - to let you know when they finish.

Sound to Play

The name of the sound file (.wav) to play. The browse button can be used to find the file. The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Default Editor (if no association found)

When a filename is double clicked within the file list or in the results pane of the status dialog, the editor specified here will be used if no association can be found for the file. The default is Windows Notepad. The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Diff tool (text difference)

Provides integration with an external text difference tool, such as WinDiff or Beyond Compare.

BeyondCompare - http://www.scootersoftware.com

WinMerge - http://winmerge.org

WinDiff - http://www.grigsoft.com/download-windiff.htm

DeltOpia - http://www.deltopia.com

Show Tips of the Day

When enabled, Tips of the Day are shown when TextPipe begins.

Language

This item allows you to change the display language used by TextPipe. To translate TextPipe into your language, or to find what translations are available, please visit http://www.datamystic.com/languages.html

Fixed Width Font

The font to be used for displaying text in the trial input and trial output areas. To change the font, click the Change button.

File History Entries

The number of recently used filters to display in the file menu.

Reopen last filter

Automatically reopens the last filter when TextPipe starts

Ok

Saves changes and closes the window.

Cancel

Discards changes and closes the window.


Tools Menu

 Tools Menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Prepare filter for distribution

This item clears the Trial Input and Output areas and the Scratch Pad area, so any data that you have been testing with won't be seen by the people who receive your filter.

Disable prompting

This item checks the filter for any form of user prompting and offers to disable it. This is very useful when you need to automate a complex filter, because it saves having to check each and every search/replace filter to see if prompting is enabled, and also checking the input filter to see if Prompt on Binary Files or Prompt Before Each File is checked.

Put sample in Trial Input area

Depending on the Input Filter's selection, either grabs a sample of text from the clipboard, or grabs the entire first file in the file list. An error message is shown if there are no files in the file list or the file cannot be found.

Move Trial Output to Trial Input

Copies the text in the Trial Output area back to the Trial Input area.

Verify filter list

This command checks all the filters in the filter list for errors. This action is always performed before a filter list is run.

wizard Filter wizard

Now found in the Filters Menu, the Filter Wizard provides a step-by-step method for beginners to construct a TextPipe filter list.

Web-based filter wizard

Now found in the Filters Menu, this filter wizard shows how easy it is to use a web page to control TextPipe to perform custom tasks.

Customize shell extensions

This option lets you customize the items that appear in the right-click menu in Windows Explorer when you click on a file, folder or drive.

Show command line

Displays the command line used to start TextPipe. This can be used to trouble-shoot command line parameters if you use macros to substitute variables.

Command line wizard

Gives examples of invoking the current filter from the command line.

Clipboard viewer

Displays the Windows clipboard viewer, which displays the current contents of the Windows clipboard. This can be very handy for testing a set of filters, because you can easily see what happens to the text, both before and after.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Notepad

Displays NotePad, a small editor useful for editing plain text files (files that contain no formatting like bold, fonts etc). This can be very handy for generating test data for a set of filters. NotePad is automatically installed with Windows.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Wordpad

Displays WordPad, a small editor useful for editing both plain text files (files that contain no formatting like bold, fonts etc) and Word documents, Rich Text Format (RTF) documents and Windows Write (WRI) documents. This can be very handy for generating test data for a set of filters.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Character map

Displays Windows Character Map. This program is usually used to insert special characters into documents, and it can also be used to determine the ASCII code of a character.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Calculator

Displays the Windows Calculator.

If you didn’t install this component of Windows, you can do so by checking it under Accessories

Show log file

Moved to Filter options window.

Truncate log file

Moved to Filter options window.


Customize shell extensions

 Customize shell extensions
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This option lets you customize the TextPipe menu items that appear in the right-click menu in Windows Explorer when you click on a file, folder or drive.

This function is found in the Tool menu\Customize shell extensions.

See also

Command line parameters

Menu name

The name given to the filter in the Windows Explorer right-click context menu.

Comment

The comment is displayed on the Windows Explorer's status line at the base of the window. It gives more information about what a filter does.

Filter file name

The name of the filter file to use. This field is optional - if you do not need to use a filter file you don't need to specify one.

Command

The TextPipe command line to use. This allows you to tailor how TextPipe is run - whether it runs and then exits immediately (using a '/g /q' on the end), etc.

%f is replaced by the filter file name (if one is specified).

%p is replaced by the list of selected files, each one is double quoted automatically

Commonly used options are:

· /g Go - run filter
· /q Quit after running filter
· /i- Do NOT descend into subfolders
· /CLEARFILES - (recommended) include this prior to %p so that any existing filenames on the Files To Process list are removed.

Note: When a filter is triggered from Windows Explorer, the command line option /fromshellext is added automatically after the filter name. This option causes TextPipe to clear any existing files from the filter's file list before adding the selected files from Windows Explorer. A warning is given if any files are cleared.

Add new rows

Move the cursor to the very last cell, then press Tab. This creates a new row below the last row.

Delete a row

Select the row by clicking it, then click the Delete Row button.

Change the order of items in the menu

Use the row selector on the left of each row to drag the row to a new position

Ok

Saves the changes and closes the window.

Apply

Saves the changes but does not close the window, allowing you to preview the changes in Explorer.

Cancel

Does not save any changes and closes the window.


Show command line

 Show command line
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This function is found in the Tools menu\Show command line.

This function is useful if you are not sure how your command line arguments are being passed to TextPipe. TextPipe does not show quoted parameters in the grid - the actual grid contents show how TextPipe has interpreted the command line. The string adds quotes back in for parameters that contain spaces - it is not an exact representation of the original command line. The text also shows the current path (PATH), and all the Environment Variables passed to TextPipe when it was started.

Environment Variables can be inserted into TextPipe using the macro @ENV_varname

See also

Command line parameters


Window menu

 Window menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Cascade

Arranges each window so that they overlap.

Maximize

Increases the size of the active window so that it fills the TextPipe window area entirely.

Minimize

Shrinks the current window to just the height of its title bar.

Minimize all

Shrinks all windows to just the height of their title bar. To restore a minimized window: If the window or another filter is showing, click the name of the document you want to restore on the Window menu.

Restore

Grows a minimized window back to its pre-minimized size, or shrinks a maximized window back to its pre-maximized size.

Arrange Icons

Arranges the title bars of the minimized windows.

Next

Changes focus to the next child window.

Previous

Changes focus to the previous child window.


Help Menu

 Help Menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Contents

Displays the contents page of the TextPipe online help.

Index

Displays the index page of the TextPipe online help.

Tutorial

Displays the beginners tutorial and introduction to TextPipe.

Show Tip of the Day

Shows a series of handy hints for using TextPipe.

Related products

Displays a list of applications and products related to TextPipe at the DataMystic website.

Buy TextPipe Now!

Opens your web browser at http://www.datamystic.com/buy.html to purchase TextPipe.

Order form

Opens a text-based order form to order TextPipe.

TextPipe Home Page

Opens your web browser at http://www.datamystic.com/textpipe.html

DataMystic Home Page

Opens your web browser at http://www.datamystic.com

Check web for TextPipe updates

Opens the Check for Updates dialog.

Subscribe to TextPipe update list

Join the TextPipe email list to receive notification of program updates and new releases. To join, send an email to

mailto textpipe-subscribe@yahoogroups.com

or go the following web page.

Subscribe to TextPipe discussion list

Join the TextPipe email list to talk to other TextPipe users and gain knowledge on how to use TextPipe better. To join, send an email to

mailto textpipe-discuss-subscribe@yahoogroups.com

or go the following web page.

Technical support

Opens your web browser at http://www.datamystic.com/support.html

Suggestion box

Opens your web browser at a form you can fill in to send feedback to DataMystic.

About

Displays TextPipe's About window.


Check for updates

 Check for updates
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Check for updates

Found in the Help menu, this option lets you check if you have the most up-to-date version. This dialog box is automatically displayed 5 days after the software is first installed.

Unless the Don't ask me again until date is set in the future, the software will check the web for updates every time it is started. If Windows is setup to automatically dial-out if no Internet connection is detected, then checking for updates may cause the Dial-Up dialog to appear.

Check web for updates now

Selecting this radio button and then clicking on Ok results in an access to the DataMystic web site to determine if your software has been superceded. If your copy is out of date, the software will confirm that you wish to download a new version. If so, it will open a web browser at a download page. If you do not wish to download, it will check again in 30 days time.

Don't ask me again until

Selecting this radio button and then setting a date in the calendar drop-down allows you to set a date after which it will again attempt to check for an update. Set this date well in the future if you don't want it to check for updates.

Initially this date is set to 5 days after the software is installed.


Advanced Topics

 Advanced Topics
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Optimizing performance, temporary files

Automation Examples

Command Line Options

Troubleshooting your filters

Troubleshooting command line automation

COM interface for scripting

Defining custom patterns

Unprintable Characters

Null characters - Clipboard Limitations

File Size Limitations imposed by the Operating System, or by some filters

Task bar menu

Logging

Exit codes

Registry entries

Data mining

Standard ASCII codes

Decimal to Hex Conversion Chart

System requirements


Task bar menu

 Task bar menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The task bar shows a small TextPipe icon while TextPipe is running.

By right clicking on the icon, you get a menu of options. The options below the line are filters that you can run.

Typically the filters process text that is on the clipboard. This is so that you can copy text from your editor or Word Processor to the clipboard, run a TextPipe filter via the taskbar icon, and then paste the converted text back into your application.

You can also process files e.g. copy a file to the clipboard (e.g. to copy your favorite mail signature to the clipboard ready for pasting).

To add your own filters to the list shown, just copy or move them to the folder <TextPipe Install Folder>\TaskMenu. for a default installation this folder is My Documents\DataMystic\TextPipe\TaskMenu.

Go

Clicking Go causes the current filter to be run (rather than a filter on the taskbar menu)

Clipboard

Clicking Go causes the current filter to be run (rather than a filter on the taskbar menu) with the input and output forced to the clipboard (much the same as the Clipboard button in the main window).

Force clipboard

This option causes any filters clicked on from the list below it to be run with the input and output forced to the clipboard (much the same as the Clipboard button in the main window), regardless of how they may be defined.

Clear clipboard

Clears the contents of the clipboard

Refresh list

Causes TextPipe to re-read the list of files in the TaskMenu folder.


Logging

 Logging
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The filter options pane can be reached by selecting the topmost node in the filter tree. In the registered version it allows you to set logging profiles. Logging allows you to track when a filter was run, who ran it, how long it ran for, which files it processed or skipped, the input size and output size of each file and the changes made to it by each filter. It also keeps track of any errors that occurred during a filter job.

There are three levels of log entry - Info, Warning and Error. Logged errors modify TextPipe's exit code.

Logging enabled

When checked, this allows log information to be sent to the log file.

Log filename

The name of the file to log to.

The filename can contain Environment Variables escaped between % signs e.g. %TEMP%\myfile.txt. TextPipe corrects any doubled backslashes.

Show

Opens the log file in the associated application for the file type, or if none exists, or if no association exists, using the default editor specified in Preferences.

Delete

Deletes the log file after prompting for confirmation.

Append to file

Adds to an existing log file if one exists

Overwrite file

Clears any existing log file before adding log data

Don't log to file if line count is less than...

No lines are written to the log file unless this threshold is exceeded. Any lines are stored in the results screen and can be transferred to the log file if desired. A value of zero prevents logging - all lines are displayed.

This option is useful for preventing logging for small jobs, and keeps the size of the log file small.

Include comment filters in log

Including filter comments in the output log can greatly assist debugging complex filters. Note that processing may be slightly slower when they are included.

See also

Command line parameters


Automation Examples

 Automation Examples
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Command line automation can be initiated in two basic ways (see here for COM automation):

1. via a batch file
2. via a Desktop or Start Menu shortcut (create a TextPipe shortcut, then change the Target property under the Short Cut tab)

Either way, you need to be careful to quote any long filenames e.g.

"C:\Program Files\DataMystic\TextPipe\TextPipe.exe" /f=filter_file.fll "c:\my documents\*.*" /g /q

You can use the command line automation helper to give you a variety of examples of automating the current filter.

This simple line starts TextPipe, opens the filter called filter_file.fll, then runs it across all files in c:\my documents\*.*. The /g option tells TextPipe to "start now", and the /q option tells TextPipe to "quit when done". Omitting either of these options or putting them in the wrong order will prevent automation from working.

A set of filtering options can also be specified using the command line (rather than using a filter file). Not all filtering options can be specified on the command line. This limitation can be avoided completely by saving a filter (.fll file) with the appropriate filtering options, and then using the /F=<filter file> option to open it from the command line.

MS-DOS Prompt START command

You can use the MS-DOS Prompt's START command to force the command shell to wait for TextPipe to finish before it continues with the next command. This can be very useful if you run a series of two or more TextPipe filters in sequence (also referred to as 'chaining' filters together). The full command line options for the Start command are:

C:\>start /?

Runs a Windows program or an MS-DOS program.

START [options] program [arg...]

START [options] document.ext

/m[inimized] Run the new program minimized (in the background).

/max[imized] Run the new program maximized (in the foreground).

/r[estored] Run the new program restored (in the foreground). [default]

/w[ait] Does not return until the other program exits.

So to run multiple TextPipe filters in a row, use a sequence like the following:

start /w "C:\Program Files\DataMystic\TextPipe\TextPipe.exe" /f=filter1.fll "c:\my documents\*.*" /g /q

start /w "c:\program files\TextPipe\TextPipe.exe" /f=filter2.fll "c:\my documents\*.*" /g /q

...


Data mining

 Data mining
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

What is Data Mining?

Data mining or text mining is where a source of information is processed to extract information. Examples include:

· Process a web site to extract product catalog and cost information. This can then be used to compare prices between different suppliers.
· Process a web site to extract email addresses or web URLs.
· Harvest the data on a web site for your own purposes.

Extracted data is designed to be easily loaded into a database for further analysis.

Click here for our white papers on Web Site Data Mining, Report Mining and more

How can TextPipe help?

TextPipe can be used to generate an extract from any text data source, including web sites. TextPipe can also be used to perform data cleansing or any additional processing e.g.

· add a header record (e.g. provide column titles for .CSV files)
· remove unwanted data
· replace specific text
· convert line feeds to DOS/Unix/Mac
· expand tabs
· fix capitalization
· convert from EBCDIC to ASCII
· remove multiple whitespace
· remove columns, lines or fields
· remove duplicate records
· sort
· extract email addresses from specific fields
· discard records matching a pattern
· and much more

Click here for more information about TextPipe

How can Offline Explorer help?

MetaProducts Offline Explorer (OE) can download your favorite Web, FTP and HTTPS sites (up to 100 simultaneously) for later offline viewing, editing or browsing (OE has a built-in internal browser). Offline Explorer allows you to selectively (include or exclude) individual servers, directories, and files using only keywords. OE is known for its excellent user interface and as one of the fastest known Web site downloaders available on the market today. Support of industry-standard technologies, like FTP, HTTPS, different proxy servers, Macromedia Flash, Cookies, XML, a built-in internal HTTP server that allows you to share downloaded files over an Intranet, makes Offline Explorer the leader in offline browsing.

For data mining applications, Offline Explorer can run TextPipe when a web site has completed downloading. OE knows which file types contain text, so it tells TextPipe to only process those files.

TextPipe can also be used to reduce the size of downloaded web sites:

· convert line feeds from DOS/Unix/Mac to None
· remove multiple whitespace
· remove whitespace at the start and end of each line
· remove blank lines
· remove binary characters

Click here for more information about Offline Explorer


COM interface (API)

 COM interface (API)
Home User Assistance Tutorials How to Use TextPipe Advanced
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

You can script TextPipe's powerful text-processing functions from any external programming language using TextPipe's COM interface. You can also use the File Menu\Export functions to generate VBScript or JScript code to create and run a given filter. A sample script is included in the VBScript\sample.vbs file. You can also see our Web page sample. To make use of this control with Windows Vista you may have to disable User Access Control (UAC).

When TextPipe runs it automatically registers itself as a server. This is also performed at install time, but you can force it using textpipe.exe /regserver or textpipe.exe /unregserver. For the TextPipe Engine DLL, run regsvr32.exe textpipeengine.dll or regsvr32.exe /u textpipeengine.dll.

Notes:

1. Before adding filters to the tree, you MUST call startFilters. When you've finished adding filters, you MUST call endFilters. Together with startSubfilters and endSubfilters these control the depth that the filters are added in the tree.
2. To create sub filters you MUST surround them with startSubfilters and endSubfilters commands.

The same interface is supported by both TextPipe.Application (the interactive application) and TextPipeEngine.Application (the ultra-high speed component, found in the separately purchased TextPipe Engine).

TextPipe.Application and TextPipeEngine.Application

Function/Method/Property

Description

newWindow() : TextPipe.FilterWindow

Returns a new TextPipe.FilterWindow object that holds a new TextPipe filter window

minimize()

Minimizes TextPipe's main window

restore()

Restores TextPipe window to its previous size

hide()

Hides TextPipe's window

show()

Shows TextPipe's window

version() : string

Returns the TextPipe version number string

findWindow( filename : string ) : TextPipe.FilterWindow

Searches for an existing window that already has the given filter filename loaded, and returns it if found. If not found, it opens a new window, loads the given filter into it and returns it.

calledFrom( applicationName : string )

Sets the calling application name. When set, the File\Exit menu item changes to 'Exit and return to applicationName', and when the user tries to close TextPipe it simply hides itself. This is very handy when calling TextPipe from a third party application.

close()

Attempts to close TextPipe, prompting the user to save any modified filters.

quit()

Closes TextPipe WITHOUT prompting the user to save any modified filters. This function is the only way to close TextPipe after it has been made visible (even if TextPipe is made invisible again)

property visible : boolean

True if the main form is visible, False if it is not. When called from COM, TextPipe's window is initially invisible. As soon as TextPipe is made visible, it locks itself in memory to prevent itself being closed by COM. The only way to close it is to call the Quit function (making it invisible does not change this)

property activeFilter : TextPipe.FilterWindow

Returns the top-most filter window. If none, returns nil.

property installationFolder : string

Returns the folder where textpipe.exe file is located, with a trailing backslash. This can be used to find the correct local path to load a pre-defined filter from.

property Application : TextPipe.Application

Returns the Application object

property Parent : TextPipe.Application

Returns the Application object

property currentDirectory : string

Gets or sets the current directory. This is used by DataPipe to force TextPipe to save filters in DataPipe's filter directory rather than in a directory assigned (randomly) by Windows.

analyzeFile( filename : string ) : string

Attempts to determine the encoding and type of the given file. Returns the same results as the File Grid right-click on file\Analyze

isBinaryFile( filename : string; sampleSize : integer ) : integer

Detects if the given file is binary or not. The sample controls how much of the file to check (default 100 bytes).

Return values:

· 0 - Unknown - The file was too small to determine if it is binary
· 1 - NotBinary - Not a binary file
· 2 - Binary - a binary file
· 3 - Error - couldn't open the file.

property Application : TextPipe.Application
property Parent : TextPipe.Application

Returns the Application object

TextPipe.FilterWindow

Open and closing files

openFilter( filename : string )

Opens a filter with the specified filename

saveFilter( filename : string )

Saves the current filter with the given filename

close()

Closes the filter window

closeWithoutSave()

Same as close but closes the window without prompting even if the filter has been changed

property filterChanged : boolean

Read/Write. True if the current filter has been saved, False if it has been changed

property scratchPad : string

Read/Write. The scratch pad text

property trialInput : string

Read/Write. The trial input text

property comments : string

Read/Write. The comments tab text.

property isRunning : boolean

Read Only. True if the filter is currently running.

terminate() : boolean

Terminates the currently running filter process (initiated with execute() or executeClipboard()). Returns true if successful, false if not.

property Application : TextPipe.Application
property Parent : TextPipe.Application

Returns the Application object

Filters

Before any filters are added or any Logging, Input or Output settings are changed, you must call

startFilters()

first, and then call

endFilters()

when you have finished adding or changing. These function turn off the GUI, making adding filters much faster. They also prevent the GUI settings from overwriting the settings you make.

To create sub filters, you must surround each set of sub filters like this:

startSubfilters()

...define filters here...

endSubfilters()

clearAllFilters()

Removes all filters from the filter list

moveCursorToEnd()

Moves the cursor in the treeview to the end of the list. By default, when a filter is loaded the cursor is on the Input Filter. If you want to add additional filters you generally will want to move the cursor to the end of the filter list first.

makeSubfilterOf( parent : treenode, child : treenode )

This function has been superseded by the startSubfilters and endSubfilters commands.

Moves the child filter so that it becomes the last child sub filters of the parent filter. You must determine if this action is legal or not yourself - TextPipe does no error checking.

addSelectionFilter( type : integer, locateMethod : integer, param1 : integer, param2 : integer, moveTo : integer, processIndividually : boolean, excludeDelimiter : boolean, excludeQuotes : boolean; delimiter : integer; customDelimiter : string; hasHeader : boolean ) : treenode

Adds a filter that specifies a range. Note that Restriction filters require sub filters in order to have any effect; Delete/remove and Move/Copy do not.

Type - the type of filter to add

0 Delete column

1 Restrict lines

2 Restrict columns

3 Restrict to bytes

4 Restrict to delimited fields (CSV, Tab, Pipe etc)

5 Restrict tab-delimited fields (deprecated, use 4 instead)

6 Remove lines

7 Remove delimited fields (CSV, Tab, Pipe etc)

8 Remove Tab-delimited fields (deprecated, use 7 instead)

9 Move columns

10 Move delimited fields (CSV, Tab, Pipe etc)

11 Move Tab-delimited fields (deprecated, use 10 instead)

12 Copy columns

13 Copy delimited fields (CSV, Tab, Pipe etc)

14 Copy Tab-delimited columns (deprecated, use 13 instead)

15 Restrict to delimited fields with column header (deprecated, use 4 instead)

16 Restrict to tab fields with column header (deprecated, use 4 instead)

17 Remove Byte Range

18 Delete pipe-delimited fields (deprecated, use 7 instead)

19 Restrict to pipe-delimited fields (deprecated, use 4 instead)

20 Restrict to pipe-delimited fields with header (deprecated, use 4 instead)

21 Move pipe-delimited fields (deprecated, use 10 instead)

22 Copy pipe-delimited fields (deprecated, use 13 instead)

Locate Method - how to determine which areas to affect

0 Restrict %d .. %d

1 Restrict %1:d starting at %0:d

2 Restrict %1:d starting at END - %0:d

3 Restrict %d .. END - %d

4 Restrict END - %d .. END - %d

param1, param2 - the required integer values for the locate method

moveTo : integer - where to move or copy the columns or fields to. Default 1.

processIndividually - whether or not to apply sub filters to each CSV or Tab field individually, or to the fields as one string value. Default false.

excludeDelimiter - whether or not to include the comma or Tab field delimiter when passing the field to the sub filter. Default true.

excludeQuotes - whether or not to include the CSV quotes that may surround the field when passing the field to the sub filter. Default true.

delimiter - (optional) the index of the standard delimiter to use, or 6 for custom, default 0 for CSV

customDelimiter - (optional) the custom delimiter to use, default blank

hasHeader - (optional) true if the file's first row is a header row, default false.

addGrepFilter( type : integer, pattern : string, includeLineNumbers : boolean, includeFilename : boolean, ignoreCase : boolean, countMatches : boolean, patternType : integer; utf8support, ignoreEmpty : boolean ) : treenode

Adds a Grep type line based filter.

type

0 Restrict lines matching

1 Restrict lines NOT matching

2 Extract matches

3 Extract matching lines (grep)

4 Extract non-matching lines (inverse grep)

5 Remove matching lines

6 Remove non-matching lines

pattern - the pattern to match

includeLineNumbers - true to include the line number where the pattern was found, default False

includeFilename - true to include the filename where the pattern was found, default False

ignoreCase - ignore case when matching the pattern, default True

countMatches - true to only output a count of the number of matches, default False

patternType

0 Perl pattern

1 Egrep pattern

2 Brief pattern

3 MS Word pattern

4 EasyPattern

utf8support - true to allow matching Unicode UTF8 characters, default False

ignoreEmpty - ignore empty matches, such as for ^\d* could also match an empty line

addSplitFilter( type : integer, splitSize : integer, splitChar : string, splitCharPos : integer, splitCharCount : integer, splitLines : integer, splitFilename : string, firstfileNumber : integer; preventOverload : boolean ) : treenode

Adds a split type filter

type

0 Split at a given size

1 Split at a given character

2 Split at a given number of lines

splitSize - the size file to split at

splitChar - the character to split at

splitCharPos -

0 Split before the character (it goes into the next file)

1 Split after the character (it remains in the first file)

2 Split on top of the character (remove it)

SplitCharCount - the number of times to see SplitChar before splitting

SplitLines - (optional) split after a given number of lines, default 60

SplitFilename - (optional) the name to give to each output split file, default %f.%e.%3.3d

firstFileNumber - (optional) the number of the first file, default 0

preventOverload - (optional) true to prevent more than 10,000 files in one folder, default false

addMergeFilter( type : integer, filename : string ) : treenode

Adds a merge type filter

type

0 Merge into filename

1 Retain lines found in filename

2 Remove lines found in filename

3 Link filter filename

filename - the filename to use

addHeadTailFilter( includeOrExclude : boolean, linesOrBytes : integer, startOrEnd : boolean, count : boolean, ) : treeNode

Adds a Head/Tail type filter

includeOrExclude - whether to include or exclude the text, False for Include, True for Exclude

linesOrBytes - measure in terms of lines or bytes, False for Lines, True for Bytes

startOrEnd - measure from the start or end of the file, False for Start, True for End

Count - the count of lines or bytes to include or exclude

addRunFilter( commandLine : string, inputFilename : string, outputFilename : string; removeOutputFiles : boolean; minimized : boolean ) : treeNode

Adds a Run External Program filter

commandLine - the command line of the program to run. Should include double quotes around long filenames, and also usually includes the input and output filename.

inputFilename - the filename that TextPipe should read from after the External Program writes to it.

outputFilename - the filename that TextPipe should write to for the External Program to read in.

removeOutputFiles - remove the output files (inputFilename and outputFilename) at the end of the job. Default true.

minimized - run the external program minimized. Not Yet implemented.

addSortFilter( type : integer, reverse : boolean, removeDuplicates : boolean, startColumn : integer, length, maxMemoryMB : integer ) : treenode

Adds a sort type filter

type - the sort type

0 ANSI sort (case insensitive)

1 ANSI sort (case sensitive)

2 ASCII sort (case insensitive)

3 ASCII sort (case sensitive)

4 Numeric sort

5 Sort by length of line

6 Sort by date and time

7 Sort by date

8 Sort by time

reverse - whether to reverse the normal sort order, when set to True the order is Descending, default false

removeDuplicates - whether to remove duplicate values or not, default true

startColumn - the column to start comparisons from

length - the length of the comparison

maxMemoryMB - the maximum memory to use for sorting, 0 for auto, for 100MB, enter '100'

addDuplicatesFilter( type : integer, ignoreCase : boolean, startColumn, length : integer; includeOne : boolean, formatResults : string ) : treenode

Adds a Show or Remove Duplicates filter

type -

0 Remove duplicate lines

1 Show duplicate lines

ignoreCase - whether to ignore case during comparisons or not

startColumn - the starting column for the comparison

length - the length of the comparison

includeOne - include lines with a count of 1

formatResults - how the output should be formatted, e.g. '%d %s' to show the count followed by the string (only for type=1)

addReplaceFilter( searchStr : string, replaceStr : string, type : integer, matchCase : boolean, wholeWord : boolean, caseSensReplace : boolean, promptOnReplace : boolean, extract : boolean, firstOnly : boolean, skipPromptIdentical : boolean; action : integer ) : treenode

Adds a search and replace (find and replace) filter. See also the setPerl, setEditDistance and setBufferSize functions (below) for setting additional parameters.

searchStr - the string to search for

replaceStr - the string to replace it with

type - the find type:

0 Replace

1 Pattern (old style)

2 Sounds like

3 Edit distance

4 Perl pattern

5 Brief pattern

6 Word pattern

7 EasyPattern

matchCase - Matches case when set to True, ignores case when set to false (default)

wholeWord - Matches whole words only when set to True, default false

caseSensReplace - Replaces with matching case when set to True, default false

promptOnReplace - Prompts before replacing when set to True, default false

extract - when True, all non-matching text is discarded, default false

firstOnly - when True, only replace the first occurrence, default False

skipPromptIdentical - when True, don't bother prompting if the replacement text is identical to the original.

action - the action to perform when found; 0=replace (default), 1=remove, 2=send to subfilter, 3=send non-matching to subfilter, 4=send subpattern 1 to subfilter etc

addReplaceListFilter( filename : string, type : integer, matchCase : boolean, wholeWord : boolean, caseSensReplace : boolean, promptOnReplace : boolean, firstOnly : boolean, skipPromptIdentical : boolean ) : treenode

Adds a search and replace list filter. See also the setPerl, setEditDistance and setBufferSize functions (below) for setting additional parameters.

filename - the file to load search/replace pairs from

Other options - as above.

setEditDistance( threshold : integer ) : boolean

Sets the edit distance threshold for the immediately preceding search/replace filter. Default 2.

setBufferSize( bufferSize : integer ) : boolean

Sets the buffer size for the immediately preceding search/replace filter. Default 4096.

setPerl( bufferSize : integer, greedy : boolean, allowComments : boolean, dotMatchesNewLines : boolean ) : boolean

Sets the perl matching options for the immediately preceding search/replace filter.

bufferSize - the maximum buffer size to use for matches. Any match must fit into this buffer, so if you want to match larger pieces of text, increase the size of this buffer to suit. Default 4096.

greedy - if the pattern finds the longest match (greedy) or the shortest match. Default false.

allowComments - allow comments in the perl pattern. Default false.

dotMatchesNewLines - allow the '.' operator to match all characters, including new lines. Default true.

addNumberFilter( type : integer, value : integer ) : treenode

Adds a Number-type filter.

type - the type of filter to add

0 Convert Tabs to Spaces

1 Convert Spaces to Tabs

2 Word wrap (column width %d)

3 Pad to width of %d

4 Centre in width of %d

5 Right justify in width of %d

6 Restrict CSV field %d

7 Restrict tab-delimited field %d

8 Truncate to width %d

9 Force to width %d

10 Repeat file %d times

11 Restrict to blocks of length

12 Expand packed decimal (with implied decimals)

13 Expand zoned decimal (with implied decimals)

14 Expand unsigned (even-length) packed decimal

15 Expand unsigned (odd-length) packed decimal

Value - the numeric value to use

addInsertFilter( insertPos : integer, insertString : string, type : integer ) : treenode

Adds an insert type filter.

insertPos - the position to insert the string

insertString - the string to insert

type - (optional) the type of filter to add-

0 Insert column (default)

1 Insert bytes

addDatabaseFilter( connectionStr : string, mode : integer, fieldDelimiter : string, textQualifier : string, generateHeader : boolean, insertTable : string, timeout : integer; prepareQuery : boolean ) : treenode

Adds a database-type filter.

connectionStr - the database connection string.

mode

0 Delimited output

1 Fixed width

2 XML

3 Insert script

fieldDelimiter - the string to use between columns

textQualifier - the string to use around string column values

generateHeader - generates header information when True

insertTable - the name of the insert table

timeout - SQL command timeout in seconds, default 30

prepareQuery - whether to prepare the query or not (for extra speed, but is less compatible), default false

addMapFilter( filename : string ) : treenode

Adds a single byte map-type filter from a file

filename - the filename containing the map

See also prepareMapFilter

prepareMapFilter() : treenode

setMap( characterIndex : integer, outputString : string )

setMap( characterIndex : integer, outputString : string )

setMap( characterIndex : integer, outputString : string )

...

Adds a single byte map-type filter and allows it to be configured by the following statements

See also addMapFilter

prepareUnicodeMapFilter( unenteredValue : integer, replaceWith : string ) : treenode

setUnicodeMap2( startRange, endRange, outputString : string )

setUnicodeMap( characterIndex : integer, outputString : string )

setUnicodeMap( characterIndex : integer, outputString : string )

setUnicodeMap( characterIndex : integer, outputString : string )

...

Adds a Unicode map-type filter and allows it to be configured by the following setUnicodeMap or setUnicodeMap2 statements.

unenteredValue:

· 0 - Pass through unchanged
· 1 - Pass to subfilter
· 2 - Replace with value (use next option to specify it)
· 3 - Remove

Replace with:

The string to replace it with, if the Replace Type is 2.

addEOLfilter( inputLineFeeds : integer, fixedLength : integer, outputLineFeeds : integer, newLineFeed : string, removeBadEOLs : boolean ) : treenode

Adds an EOL (end of line) conversion filter

inputLineFeeds - the type of incoming line feeds

0 Unix

1 Mac

2 DOS

3 Auto

4 Fixed (use the fixedLength parameter to specify the length)

fixedLength - the length of fixedLength input files when option #4 is chosen above.

outputLineFeeds - the type of outgoing line feeds

0 Unix

1 Mac

2 DOS

3 None

4 String (use the newLineFeed parameter to specify the new string)

newLineFeed - the new line feed string on output when option #4 is chosen.

removeBadEOLs - whether to remove bad EOLs or not

addSimpleFilter( type : integer; isUnicode : boolean ) : treenode

Adds a simple filter type, requiring no special parameters.

type - the type of filter to add.

1 Convert ASCII to EBCDIC

2 Convert EBCDIC to ASCII

3 Convert ANSI to OEM

4 Convert OEM to ANSI

5 Convert to UPPERCASE (*)

6 Convert to lowercase (*)

7 Convert to Title Case (*)

8 Convert to Sentence Case

9 Convert to tOGGLE cASE

10 Remove blank lines

11 Remove blanks from End of Line

12 Remove blanks from Start of Line

13 Remove binary characters

14 Remove ANSI codes

15 Convert IBM drawing characters

16 Remove HTML and SGML

17 Remove backspaces

18 Resolve backspaces

19 Remove multiple whitespace

20 UUEncode

21 Hex Encode

22 Hex Decode

23 MIME Encode (Base 64)

24 MIME Decode (Base 64)

25 MIME Encode (Quoted printable)

26 MIME Decode (Quoted printable)

27 UUDecode

28 Extract email addresses

29 Unscramble (ROT13)

30 Hex dump

32 XXEncode

33 XXDecode

34 Reverse line order

35 Remove email headers

36 Decimal dump

37 HTTP Encode

38 HTTP Decode

39 Randomize lines

40 Create word list

41 Reverse each line

42 Convert to RanDOm case

43 Extract URLs

44 ANSI to Unicode

45 Unicode to ANSI

46 Display debug window

47 Word concordance

48 Delete all

49 Restrict to each line in turn

50 Convert CSV to Tab-delimited

51 Convert CSV to XML' )

52 Convert Tab-delimited to CSV

53 Convert Tab-delimited to XML

54 Convert CSV (with column headers) to XML

55 Convert Tab-delimited (with column headers) to XML

56 Convert CSV (with column headers) to Tab-delimited

57 Convert Tab-delimited (with column headers) to CSV

58 Restrict to file name

59 Convert Word documents to text (*)

60 Swap UTF-16 word order

61 Swap UTF-32 word order

62 Remove BOM (Byte Order Mark)

63 Make Big Endian

64 Make Little Endian

65 Compress to Packed Decimal

66 Compress to Zoned Decimal

67 Expand Binary Number to EBCDIC

68 Expand Binary Number to ASCII

69 NFC - Canonical Decomposition, followed by Canonical Composition

70 NFD - Canonical Decomposition

71 NFKD - Compatibility Decomposition

72 NFKC - Compatibility Decomposition, followed by Canonical Composition

73 Decompose

74 Compose

75 Convert Numeric HTML/XML Entities to text

76 Convert PDF documents to text

77 Restrict to ANSI files

78 Restrict to Unicode UTF16 files

79 Restrict to Unicode UTF32 files

80 Convert Excel spreadsheets to text (*)

81 Shred file

82 Unicode to Escaped ASCII

83 Restrict to Unicode Files

84 T-filter

85 Convert HTML/XML Entities to text

isUnicode - for those filters that support it (show above with a *), this flag indicates that the filter will be dealing with Unicode data. Default false.

addStringFilter( type : integer, str : string; matchCase : boolean ) : treenode

addStringFilterFile( type : integer, filename : string; matchCase : boolean ) : treenode

Adds a string-type filter.

type - the type of filter to add

0 Add left margin

1 Add header

2 Add footer

3 Add right margin

4 Remove lines that match exactly

5 Retain lines that match exactly

6 Remove lines matching perl pattern

7 Retain lines matching perl pattern

8 Add text side by side

9 Add repeating text side by side

10 Not Used

11 Not Used

12 XSLT transform

13 Restrict to lines from list

14 Restrict to lines NOT in list

15 Restrict to lines matching perl pattern

16 Restrict to lines NOT matching perl pattern

17 Restrict to filenames matching perl pattern

18 Restrict to filenames NOT matching perl pattern

str - the string to use

filename - the filename to use (where appropriate)

matchCase - case sensitive or not (where appropriate)

addLineNumberFilter( startNumber : integer, increment : integer, skipBlankIncrement : boolean, dontNumberBlank : boolean, numberFormat : string; DontReset : boolean ) : treenode

Adds a Line Number filter

startNumber - the starting line number

increment - the amount to add for each new line number

skipBlankIncrement - don't increase the line number for blank lines

dontNumberBlank - don't put a line number on blank lines

numberFormat - the format to use for the line number

dontReset - don't reset the count at the end of the file

addScriptFilter( language : string, script : string, timeout : integer ) : treenode

Adds an ActiveX script filter

language - the language of the script

script - the code

timeout - the command timeout in seconds, default 30

addCommentFilter( comment : string ) : treenode

Adds a comment type filter

comment - the comment.

addMathsFilter( operation : integer; operand : integer ) : treenode

Adds a maths type filter

operation - the operation to perform

0 +

1 -

2 *

3 div (the remainder is ignored)

4 mod (the remainder after division)

5 xor

6 and

7 or

8 not

9 shift left (0 inserted)

10 shift right (0 inserted)

11 rotate left

12 rotate right

operand - the operand to use

addXMLFilter( findType : integer; tag : string; attribute : string; endTag : string; includeText : boolean; includeQuotes : boolean; matchCase : boolean; bufferSize : integer ) : treenode

Adds an HTML/XML filter

findType - the operation to perform

0 restrict to an element

1 restrict to an attribute

2 restrict to between tags

tag - the element or start tag to find

attribute - the attribute to find

endTag - the endTag to find

includeText - whether to include the find string in the restriction result (default false)

includeQuotes - whether to include surrounding quotes in the attribute result or not (default false)

matchCase - match case exactly or not (default false)

bufferSize - the maximum expected size of the match (default 32768)

addUnicodeConversionFilter( toUnicode : boolean; conversion : integer; errorChar : char) : treenode

Adds a Unicode conversion filter

toUnicode - true to convert TO Unicode, false to convert FROM Unicode

conversion - the conversion to perform, from 1 to 144 or so. You can determine the correct value by exporting a filter that contains the conversion filter you require.

errorChar (optional) - the character to output if there is no match in the destination encoding, default space.

addUnicodeConversionFilter2( convertFrom, convertTo : string; errorChar : char) : treenode

Adds a Unicode conversion filter

convertFrom - the string description of the Input encoding e.g. 'UTF-16'

convertTo - the string description of the Output encoding. e.g 'UTF-8'

errorChar (optional) - the character to output if there is no match in the destination encoding, default space.

addCaptureFilter( varName : string; initial : string; reset : boolean) : treenode

Adds a Capture Text to Global variable filter

varName - the name of the variable

initial - the initial value of the variable, default blank

reset - true if the variable value is reset to the initial value when a new file is started.

mainframeCopybook( copybook : string; outputFormat: Integer; overlapComp, trimFields, onlyNumeric,

comp3canBeUnsigned, removeFiller, header: Boolean ) : boolean

Uses the string copybook to generate a set of filters for parsing the given mainframe copybook.

outputFormat - 0 for fixed width, 1 for CSV, 2 for Tab, 3 for Pipe delimited, default 1.

overlapComp - 1 to overlap even length comp fields, 0 to not overlap

trimFields - trim spaces and zeroes from fields

onlyNumeric - only convert numeric field

comp3canBeUnsigned - 1 if the COMP type should be treated as unsigned packed numeric (default false)

removeFiller - remove fields called FILLER*

header - add a field name header

addMainframeCopybookFilter( copybook : string; outputFormat: Integer; overlapComp, trimFields, onlyNumeric,

comp3canBeUnsigned, removeFiller, header, forceQuotes : Boolean ) : treenode

Uses the string copybook to insert a Mainframe Copybook Filter

outputFormat - 0 for fixed width, 1 for CSV, 2 for Tab, 3 for Pipe delimited, default 1.

overlapComp - 1 to overlap even length comp fields, 0 to not overlap

trimFields - trim spaces and zeroes from fields

onlyNumeric - only convert numeric field

comp3canBeUnsigned - 1 if the COMP type should be treated as unsigned packed numeric (default false)

removeFiller - remove fields called FILLER*

header - add a field name header

forceQuotes - When true, all alpha values are quoted. When false, only alpha values with embedded quotes or the delimiter are quoted

Output Filter

property outputMode : integer

Read/Write. Sets the output mode:

0 Output to clipboard (all files are joined together)

1 Output to files

2 Output merged (joined) to a single file

property outputTestMode : integer

Read/Write. Sets the output filter test mode.

0 Normal mode

1 test mode

2 Backup mode.

property outputFolder : string

Read/Write. Sets the output filter folder.

property outputRetainDate : boolean

Read/Write. Sets the output filter retain date setting. True turns on the retain date setting, default False.

property outputMergeFile : string

Read/Write. Sets the output filter to merge data to a single file, using the given filename.

property outputExtension : string

Read/Write. Sets the output filter extension.

property outputAppend : boolean

Read/Write. Sets append mode when True, default False.

property outputOnlyOutputChangedFiles : integer

0 - Always output

1 - Only output modified files

2 - Delete original if modified

property outputKeepStructure : boolean

Read/Write. When used in conjunction with outputFolder, determines if the original folder structure is retained, default False.

function addSecondaryOutputFilter( mode : integer, testMode : integer, retainDate : boolean, append : boolean, onlyOutputChangedFiles : boolean, folder : string, extension : string, keepStructure : boolean, mergeFilename : string; ignoreStartFolder, openOutputOnCompletion, removeEmpty : boolean ) : treenode

Adds a secondary output filter, usually used for splitting some lines or records to a new file.

function outputFileCount : integer

Returns the count of the number of output files

function outputFile( index : integer )

Returns output filename N, from 0 to outputFileCount-1

procedure outputFilesTask( task : integer )

Task:

1 - Deletes the output files (useful after test mode)

2 - Restores .bak files

property IgnoreStartFolder : boolean

When OutputKeepStructure is true, ignores the start folder when generating the target folder name.

property MoveModified : boolean

When the OutputFolder is different to the input folder, modified files are moved to the output folder and the original file is deleted; unmodified files are left in the original folder.

property outputOpenOutputOnCompletion : boolean

Opens each output file in its associated program on completion.

property outputRemoveEmpty : boolean

Removes empty (zero-byte) output files - useful in conjunction with the Split Files filter

Input Filter

property InputMode : integer

Read/Write. Sets the input mode:

0 Input from clipboard

1 Input from files (default). You can add files using the addFile function.

2 Input from Trial Run area. You can set the Trial Run text using the trialInput property

3 Input from random file generator. You can set the

property inputClipboardUnicode : boolean

Read/Write. In clipboard mode, controls if input from the clipboard is handled as ANSI or Unicode. Default False.

property inputBinaryFiles : integer

Read/Write. Controls how binary files are handled

value (default 0)

0 Binary files are processed

1 Binary files are skipped

2 Binary files are confirmed before processing

property InputBinarySampleSize : integer

Read/Write. Default 100. Controls the binary sample size for Input files

property InputPromptOnEach : boolean

Read/Write. Default False. Controls whether to prompt before processing each file.

property InputPromptOnReadOnly : boolean

Read/Write. Default False. Controls whether to prompt before processing each read only file.

property InputDeleteFiles : boolean

Read/Write. Default False. Controls if input files are deleted after processing. Use with caution!

property inputRandom : integer

Read/Write. Sets the input to come from a randomly generated file of size size, default 1024

Logging functions

property logEnabled : boolean

Read/Write. When True, logging is on, default False

property logFilename : string

Read/Write. The file to log data to

property logAppend : boolean

Read/Write. When True, logging appends to the existing file

property logThreshold : integer

Read/Write. When more than threshold lines are present, all get written to the log file (this is for interactive mode only, for non-interactive mode ALL lines get written to the log file)

Running Filters

execute() : boolean

Runs the current filter (same as the 'Go' button). This function does not return until the thread completes.

Returns true on success, False on failure (with error message in errorText)

executeClipboard() : boolean

Runs the current filter with input from and output to the Clipboard (same as the 'Clipboard' button). This function does not return until the thread completes.

Returns true on success, False on failure (with error message in errorText)

processString( inputText : string ) : string

Processes the string inputText and returns the result string, just as though the entire string was in a file.

If an error occurs, errorText will be set to the error message, or blank if there is no error.

Note - this function does not trigger any logging

compileFilter() : string

Compiles the filter into an internal form and saves it so that the compiling doesn't have to be repeated each time text is processed. This function is ONLY used in conjunction with processString(s)WithCompiledFilter - it does not speed up any other process. Any changes made to the filter list after the filter was compiled will not be reflected. Also, the normal Input and Output filters are ignored - a string input and output filter is used instead.

Returns a blank string on success, error message on failure. errorText is also cleared if there is no error, or set if there is an error.

processStringWithCompiledFilter( inputText : string ) : string

Processes the string inputText using the pre-compiled filter. If the filter list has not been compiled with compileFilter this function will return an empty string. This function calls startJob and endJob at the beginning and end of processing. Note the singular - String

Note - this function does not trigger any logging

compiledFilterStartJob()

processStringsWithCompiledFilter( inputText : string ) : string

...

(usually called multiple times)

...

compiledFilterEndJob()

Processes the string inputText using the pre-compiled filter. If the filter list has not been compiled with compileFilter this function will return an empty string. This function DOES NOT call startJob and endJob at the beginning and end of processing - they must be called separately. This function is normally called a number of times. Note the plural - Strings

compiledFilterStartJob() - this function prepares a compiled filter for processing.

compiledFilterEndJob() - this function completes a compiled filter's processing.

isRunning() : boolean

Returns true if a processing thread is running. Only works in conjunction with execute() and executeClipboard()

property errorText : string

Returns the text of the last error, or blank if there was no last error.

Adding Files

clearAllFiles()

Clears all files from the file list

addFile( fileSpec : string, subfolders : integer, action : integer )

Adds a file to the file list.

fileSpec - the filename, folder or wildcard

subfolders - how many subfolders to include (default 255)

0 No subfolders

1..9 1..9 Subfolder(s)

255 All subfolders

action - the action column (default 1)

1 Include the files

2 Exclude the files

3 Ignore the files

addFileList( filename : string )

Adds a list of files to the file list.

filename - the list of files is contained in this file.


COM examples

 COM examples
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The following demonstrations show how TextPipe can be controlled from a web page using VBScript or JScript. You could easily customise this page with your own logo, instructions, links to your website and much more, and then deliver it to your clients as a custom front-end. The script can also be encrypted so that users can't change it. We can also provide installation support so that you can package the front-end and TextPipe Pro for your clients.

Demo 1 - Add header/footer/left & right margins

Input data

[TTextAreaFormControlObj]

Add this header

[TTextAreaFormControlObj]

Add this left margin

[TTextAreaFormControlObj]

Add this right margin

[TTextAreaFormControlObj]

Add this footer

[TTextAreaFormControlObj]

(Output data is shown below)

Output data

[TTextAreaFormControlObj]

Demo 2 - Add filters, leave TP running

This demonstration starts TextPipe, adds a dummy filter, and then leaves TextPipe open for running.

Source code

Below is the source code embedded in this page:

<script language="VBScript">

' addStringFilter type - the type of filter to add

stftAddLeftMargin = 0

stftAddHeader = 1

stftAddFooter = 2

stftAddRightMargin = 3

sftConvertToRandomCase = 42

dim TextPipeApp2

function test()

dim Filter1

dim TextPipeApp

Set TextPipeApp = CreateObject("TextPipe.Application")

Set Filter1 = TextPipeApp.newWindow

Filter1.startFilters

Filter1.clearAllFilters

Filter1.addCommentFilter( "This filter list was created from VBScript on a webpage" )

if form1.leftmargin.value <> "" then a = Filter1.addStringFilter( stftAddLeftMargin, form1.leftmargin.value )

if form1.rightmargin.value <> "" then a = Filter1.addStringFilter( stftAddRightMargin, form1.rightmargin.value )

if form1.header.value <> "" then a = Filter1.addStringFilter( stftAddHeader, form1.header.value )

if form1.footer.value <> "" then a = Filter1.addStringFilter( stftAddFooter, form1.footer.value )

Filter1.endFilters

form1.output.value = Filter1.processString( form1.input.value )

'close TextPipe

'if you don't want to close TextPipe, omit the closeWithoutSave line

filter1.closeWithoutSave

Set Filter1 = Nothing

Set TextPipeApp = Nothing

end function

function test2()

dim Filter1

Set TextPipeApp2 = CreateObject("TextPipe.Application")

Set Filter1 = TextPipeApp2.newWindow

Filter1.startFilters

Filter1.clearAllFilters

Filter1.addCommentFilter( "This filter list was created from VBScript on a webpage" )

Filter1.addSimpleFilter( sftConvertToRandomCase ) 'randomize case

Filter1.endFilters

end function

function test3()

Set TextPipeApp2 = Nothing

end function

</script>


Command Line Options

 Command Line Options
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe can be run from a DOS shell or from a shortcut with parameters to make it run completely automatically. This is very useful for performing conversions automatically or on machine start up (by making the short cut an entry in the Start Up folder). You can also use the command line wizard to provide examples. If you are having difficulties, please see the Troubleshooting command-line automation page. TextPipe sets an exit code on completion.

Note:

1. Most commands change settings or add filters to the current filter. If you load a filter with /F=, the /F= command must appear FIRST, and any commands that modify it MUST come AFTER, otherwise the loaded filter's settings will override prior commands.
2. Any commands with embedded spaces must be surrounded with double quotes. You must surround the entire command/setting combination e.g. "/f=My Documents\DataMystic\TextPipe\My Filters\test.fll" NOT /f="My Documents\DataMystic\TextPipe\My Filters\test.fll"
3. Windows limits the command line length to about 260 characters. You can overcome this limitation by using the /Z=<filename> command to load command line parameters stored in a file. Note that you MUST remove any double quotes from around commands with embedded spaces when you put them in the command file.
4. If you want to add filters to a command-line loaded filter at run time, you must ensure that the very last filter (prior to the output filter) is not a sub filter, otherwise all added filters will be added as sub filters.
5. Not all filters or options can be set via the command line. To workaround this limitation, save a filter with your desired settings (e.g. an output base directory of 'c:\temp'), and then open the filter as the first item on the command line (using /f=<filter name>). If you use /F= a second or third time, these filters will be appended to the existing filter.
6. If you want TextPipe to start minimized, use /MINIMIZED as the very FIRST command line parameter, or make sure that it is on the command line and NOT in the command file.

You can automatically generate command line text from your filter using the File Menu\Export\Export command line to clipboard feature.

Conventions used below:

<> surrounds text that is to be entered. Do NOT enter the '<' and '>' characters

Alternatives are shown in square brackets '[' and ']' with a pipe '|' character between them. Do NOT enter the square brackets or the pipe character - just type the single alternative you require.

Usage:

"C:\Program Files\DataMystic\TextPipe\textpipe.exe" [/MINIMIZED|/MAXIMIZED] ["/F=my filter file.fll"] [commands] [wildcards] [/G [/Q]]

Where [commands] is zero or more of:

<wildcard>

A wildcard or filename to add to the list of files to process e.g.

"c:\Documents and Settings\Simon\Desktop\*.txt;*.ans"

@<name>=<value>

Predefines the value of the global variable name to the value value

/{=<string>

Adds a comment. See comment filters.

/A[S|F|T|E]=<string>

Adds a string:

· ‘S’ Adds a left margin to every line (like email quotes etc)
· ‘F’ Adds a right margin to every line
· ‘T’ Adds a header to the start of every file (e.g. banner advertising)
· ‘E’ Adds a footer to the end of every file (e.g. copyright notices)
· '1' Remove lines from list
· '2' Retain lines from list
· '3' Remove lines matching perl pattern
· '4' Retain lines matching perl pattern
· '5' Add text side by side
· '6' Add repeating text side by side

/AL=start,increment

Add line numbers, starting from start and with the given increment e.g.

/AL=500,50

See Line number filters.

/B<C|P|S>

Skip binary files. Default: Skip. Use

· 'P' for Process all binary files
· 'S' for Skip binary files
· 'C' to Confirm each binary file

This flag is reset to the default after a /G. See Input filter.

/C<U|L|S|T|I|R...>

Case translation, conversion, email and miscellaneous:

· ‘U’ for UPPERCASE
· ‘L’ for lowercase
· ‘S’ for Sentence case
· ‘T’ for Title Case
· ‘I’ for tOGGLE cASE
· 'R' for RanDOm caSE
·
· 'M' for Convert IBM drawing characters
· 'D' for debug filter
· '-' for Delete All
· 'Z' for Randomize lines
· 'W' for Word list
· 'V' for Reverse each line
·
· '0' Extract email addresses
· '1' UUencode
· '2' UUdecode
· '3' XXencode
· '4' XXdecode
· '5' Mime Base 64 encode
· '6' Mime Base 64 decode
· '7' Mime quoted printable encode
· '8' Mime quoted printable decode
· '9' Extract URLs

/CLEARCOMMENTS

Clears all comments from the filter.

/CLEARFILES

Clears all files from the file list.

/CLEARFILTERS

Clears all filters from the filter list (this does not close the filter)

/CLOSEONCOMPLETION

Closes the window for a processing thread as soon as the thread completes. This option needs to precede each /G command.

/D[A|B|D|E|H|K|L|R|S|W]...

Removes:

· 'L' Removes blank lines
· 'E' Removes all spaces and tabs from the end of lines
· 'S' Removes all spaces and tabs from the start of lines
· 'B' Removes binary characters
· ‘A’ Removes ANSI codes
· 'D' Translate IBM drawing characters
· 'H' Remove HTML and SGML
· 'K' Remove backspaces
· 'R' Resolve backspaces
· 'W' Removes multiple whitespace
· 'M' Removes email headers

/deletelistfile

Deletes the list file. Only works when placed immediately after a /L=listfile directive.

/EXPORTFILTERTOLOG

Exports the filter's text to the log file. Useful for command line automation to ensure that the filter you build via the command line is the filter you get. The log filename MUST have already been specified. Should only be used after the filter list has been completely built, otherwise you will only get partial results.

/EXTENSION=<new extension>

Sets the new file extension to be given to all output files. A blank value disables the setting. See Output filter.

/E[R]<D|M|U|A|F<cpr>><D|M|U|N>

Line feed conversion. See End of line conversion filter.

Optional R to remove extraneous line feeds.

Input format:

· ‘M’ for Macintosh
· ‘D’ for DOS
· ‘U’ for Unix/Amiga
· ‘A’ for Auto
· ‘F’ for Fixed Length, followed by the number of characters per record (cpr).

Output format:

· ‘M’ for Macintosh
· ‘D’ for DOS
· ‘U’ for Unix/Amiga
· ‘N’ for None (ie remove line feeds)

e.g.

/ERDU to convert from Dos to Unix EOL, removing extra linefeeds

/EF742D to convert from fixed record of 742 characters to DOS

/F=<filter filename>

Open filter list file.

This command opens a filter file. Opening a filter file DELETES any filters and overrides all options that have been specified by any previous command line parameters e.g.

textpipe.exe /ERDU "/f=My Documents\DataMystic\TextPipe\My Filters\test.fll"

If you have already loaded a filter file but have not yet activated it with /G, any subsequent /F commands will APPEND its filters to the existing filter list. This can be used to combine multiple filters together, even those that have sub filters. Note that this is different behaviour to inserting using the GUI, which always occurs at the current cursor position.

/FROMSHELLEXT

When a filter is triggered from Windows Explorer, the command line option /fromshellext is added automatically after the filter name. This option causes TextPipe to clear any existing files from the filter's file list before adding the selected files from Windows Explorer. A warning is given if any files are cleared.

/G

Go! Begin a conversion thread with the specified options. You can specify extra options and follow them with /G again to start a second thread of execution. After starting a thread of execution, all options return to their default settings.

After a thread has been initiated with /G, the following actions occur (in ADDITION to all flags returning to their defaults):

· The entire list of filters is cleared
· The entire list of files is cleared

Note: The /G command must be on the right hand side of all the commands that make up a thread of filtering.

/H<A|O>

Character set translation

· A for ANSI to OEM
· O for OEM to ANSI
· N for ANSI to Unicode
· C for Unicode to ANSI

/I

/I+

/I-

Subfolder recursion. Default: On. Specifying /I toggles it, /I+ turns it on, /I- turns it off.

This option is NOT reset after a /G

/L=<file list filename>

Load a list file containing a list of filenames and wildcards. The file should contain one file specification per line e.g.

C:\docs\*.txt;*.ans

D:\unix\*.*

...etc

Note: The filenames in the list file must NOT be surrounded by double quotes.

See also the /deletelistfile directive.

/LOG=<filename>

Sends log information to the specified file. This is primarily used during command line automation. If the file cannot be written to or disk space fills up logging is turned off. TextPipe will force the creation of folders if necessary. See logging options.

The default file is 'textpipe.log' in the folder TextPipe was started from. A blank log filename turns logging off, any other value turns logging on.

/LOGENABLE=(0|1)

Controls whether logging is on or off

· 0 = Logging disabled
· 1 - Logging enabled

See logging options.

/LOGMODE=(0|1)

Controls the logging mode.

· 0 = Overwrite existing file
· 1 - Append to existing file

See logging options.

/LOGTHRESHOLD=<minimum lines>

Controls the minimum number of log lines before the log data is written to the log file. A value of zero forces all lines to be written out. This is useful for preventing logging for small jobs, and keeps the size of the log file small.

See logging options.

/M=<map filename>

Map - adds a filter with the specified (single byte) map file. See Maps. See also /UNICODEMAP

/MAXIMIZED

TextPipe starts maximized.

This command line option MUST be placed FIRST.

/MERGETOFILE=<merge filename>

Sets the output filter to merge all output to a single file, and sets the filename to the filename specified. See Output filter.

/MINIMIZED

TextPipe starts minimized. TextPipe also uses less memory when minimized because the screen representation does not need to be stored. This command line option MUST be placed FIRST.

/MULTIMAP=<file.xls>

Specifies the Excel document to load as a Unicode multi-map. By setting the MULTIMAPORDER, the same Excel spreadsheet can be used to map to different output formats.

/MULTIMAPLANGUAGECALL=<string>

Specifies the format of the text string used to indicate a change of output language in a multi-map. The default is

((=%s=))

where %s is the new language name.

/MULTIMAPORDER=<Primary>(,secondary,etc)

Specifies the precedence order given to the columns of an Unicode multi-map. The value is a comma-separated list of column names (each name is stored at the top of the column).

The first column specified is called the primary. If only the primary is specified, then the remaining columns are used in left-to-right order (ignoring the primary column). If two or more columns are specified, then only those columns are searched for values.

/N

/N+

/N-

Retain the original file’s date and time. Default: Off. Specifying /N toggles it, /N+ turns it on, /N- turns if off. See Output filter.

This flag is reset to the default after a /G

/NEWFILTER=<new filter filename>

Similar to /F, but opens a new filter and sets its filename before it gets saved.

/OUTPUTFOLDER=<new output folder>

Sets the new output folder for all files to be written to. See Output filter.

/OUTPUTKEEPSTRUCT

/OUTPUTKEEPSTRUCT+

/OUTPUTKEEPSTRUCT-

Controls whether the folder structure of the input files is retained when the output files get written to a new folder. See /OUTPUTFOLDER, and the Output filter.

/P<A|E,,,>

Representation translation and other miscellaneous filters

· A for ASCII to EBCDIC
· E for EBCDIC to ASCII
· D for decimal dump
· H for hex dump
· U for ROT13 encode/decode
· 0 for hex encode
· 1 for hex decode
· 2 for HTTP encode
· 3 for HTTP decode
· 4 for reverse line order

/PLAYSOUND=(0|1)

Controls whether a sound is played on completion

· 0 = Sound is disabled
· 1 - Sound is enabled (default)

/PREPAREFILTERFORDISTRIBUTION

Prepares the filter for distribution. See the Tools Menu

/Q

/Q=milliseconds

Exit after completion, with optional delay. Default: Off. Using /Q causes TextPipe to exit 5 seconds (5000 milliseconds) after all conversion threads complete. You can optionally specify a millisecond value e.g. 500 for 0.5 seconds.

If an error occurs, it is displayed.

Note: The /Q command should be the very last command on the command line.

/R[C][P]=<replace string>

Replace string. See Search and Replace filter.

Optional C for Case Sensitive replace. Default: Off. Select this option to use the same case for the replace string as was found in the search string.

Optional P to prompt on replace. Default Off.

Must be preceded by a /S to specify the search string. After the replace string has been specified, the Case Sensitive and Whole Words options both reset to Off.

/REGSERVER

Forces registration of TextPipe's COM server - this only succeeds if the user is an administrator.

/REPLACELIST=<filename>

Loads a search/replace list filter with the specified file.

/S[C][W][X][0-6]=<search string>

Search string. See Search and Replace filter.

You can also add a list of search and replace pairs through the user interface. See also Converting Search/Replaces from other sources.

Optional C for Case sensitive search/replace. Default: Off. Select this command to search for text that is capitalized exactly the way you type it in the Search text box. If you do not select Case, TextPipe finds all occurrences of the text, regardless of capitalization.

Optional W for whole words only (Default: Off)

Optional X to specify that the search string is a regular expression. Default off. This is equivalent to /S1, the old egrep-style pattern.

0-6 Search type. Default 0.

· 0 - Exact search
· 1 - Old egrep pattern
· 2 - Sounds like
· 3 - Edit distance
· 4 - Perl pattern
· 5 - Brief-style pattern
· 6 - Word-style pattern
· 7 - EasyPattern
· 8 - Unicode (UTF-16LE)
· 9 - Unicode pattern (UTF-16LE)

/S... is used in conjunction with the /R command to specify the replace string. For strings with embedded spaces, surround the whole parameter in quotes e.g.

textpipe.exe "/SC=roses are blue" "/R=violets are yellow"

/SAVEFILTER=<filter filename>

Saves the filter to the given filename. Any existing file is overwritten. The filename is added to the list of recent filters. TextPipe will force the creation of folders if necessary.

/ST<spaces per tab>

Compresses spaces to tabs. See Spaces to Tabs filter.

/T

/T+

/T-

/TB

Test mode. Default: Off. Specifying /T toggles it, /T+ turns it on, /T- turns if off, /TB enables backup mode. See Output filter.

This flag is reset to the default after a /G

/TS<spaces per tab>

Expands tabs to spaces. Each tab is replaced by from 0 to Spaces Per Tab-1 spaces depending on the current column. See Tabs to Spaces filter.

/UNICODEMAP=<unicode map file1;file2;file3>

Adds a filter with the specified (unicode) map file. Maps can be loaded from Excel worksheets (.XLS), Comma Separated Value files (.CSV, the default when the file extension is not recognized) and Tab delimited value files (.TAB). This command should be followed by the /UNICODEMAPOPTIONS command below, otherwise non-entered characters are passed through unchanged (Replace Type 0).

Multiple maps can be separated by semi-colons (;)

See Unicode maps. See also /M

/UNICODEMAPAPPEND=<unicode map filename>

Appends a map file to the previously added Unicode Map. This command does NOT accept multiple map filenames.

/UNICODEMAPERRORS=<0|1>

This command determines how overlapping regions are handled for the previously added Unicode map. When set to 0, overlapping regions are flagged as warnings (the default). When set to 1, overlapping regions are logged as errors.

/UNICODEMAPOPTIONS=<type>,<replace with>

Sets the non-entered character options for the previously added Unicode Map filter.

Replace Type:

· 0 - Pass through unchanged
· 1 - Pass to sub filter
· 2 - Replace with value (use next option to specify it)
· 3 - Remove

Replace with:

The string to replace it with, if the Replace Type is 2.

e.g.

/UNICODEMAP=my_translations.xls

/UNICODEMAPOPTIONS=2,Invalid character found 0x%4.4x

See Unicode maps.

/UNICODEMAPUTFMODE=<0|1>

Sets the UTF mode for the previously added Unicode Map filter.

· 0 - UTF-16
· 1 - UTF-32

See Unicode maps.

/UNREGSERVER

Forces unregistration of TextPipe's COM server - this only succeeds if the user is an administrator.

/WW<column width>

Word wrap. See Word Wrap filter.

/Z=<filename>

Read command line parameters from a command file. Very useful for creating filters based on Perl scripts or other search/replace programs, or for working around the Windows limitation of approx 260 characters on the command line. Note that you MUST remove the double quotes around command when you place them inside the command file, AND place ONE command per line. This option ignores blank lines in the command file, and allows comments on lines starting with a ';'.

For example, if you had the long command line

start /w textpipe.exe "/f=my special filter.fll" /g "/f=my special filter2.fll" /g "/f=my special filter3.fll" /g "/f=my special filter4.fll" /g /q

This would change to something like

start /w textpipe.exe "/z=commands.txt"

and the file commands.txt would contain

;my new command file

/f=my special filter.fll

/g

/f=my special filter2.fll

/g

/f=my special filter3.fll

/g

/f=my special filter4.fll

/g

;exit on completion

/q


Exit codes

 Exit codes
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe sets an exit code on completion that depends on the level of errors logged. Exit codes are useful for detecting program success from command line execution. You can use the DOS ERRORLEVEL command in a batch file to modify the execution flow according the exit code.

Highest logged error | Exit code
None | 0
Info | 1
Warning | 2
Error | 3
Abnormal termination - e.g. due to process being killed or unexpected error | 255
Stack overflow error - usually caused by a pattern match with unbounded quantifiers (where huge numbers of unnecessary matches are done), see Possessive quantifiers | 254 or 1073741571

e.g. in a .BAT batch file.

textpipe.exe ....

IF ERRORLEVEL 3 do_something

IF ERRORLEVEL 2 do_something

IF ERRORLEVEL 1 do_something


Troubleshooting

 Troubleshooting
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This section describes how to troubleshoot command line automation.

1. Check that /g and /q are at the VERY END of the command line. They affect all parameters specified to their left
2. Verify that you have used EXACTLY the right format for the parameters - see the reference section
3. Try removing the /g and /q and -
3.1. Check that the correct filter file (.fll) is opened or that the correct list of filters appears
3.2. Check that the correct files are displayed in the file list. Note that any incorrect, unrecognized or otherwise invalid command line parameters will be listed as filenames
3.3. If no files are being processed - check that the Process Binary Files option is set correctly - the files may be being detected as binary files
3.4. Check that the correct operation is performed when you click the Go button.


libiconv

 libiconv
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

About libiconv

libiconv is the GNU library used by TextPipe for its Unicode conversions.

C-Source code, .obj files and binaries are available for free from

http://www.gnu.org/software/libiconv/


System requirements

 System requirements
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe requires Windows 95/98/ME/NT/2000/XP/2003/Vista and a minimum of 16MB of main memory. The program requires approximately 15 MB of memory when running. A full installation requires 10 MB disk space.


File size limitations

 File size limitations
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe has NO upper limit on the size of files being processed - even files larger than 2GB can be processed or generated. Currently, only the Reverse line order filter requires the entire input file to be read into memory.

Several filters also require an entire ancillary file to be completed loaded into memory:

· Retain lines found in
· Remove lines found in

If you need to use these filters, you should try to keep the maximum file size to less than the total amount of physical RAM in your system. This ensures that system performance is still reasonable. For example, a system with 32 MB RAM should be Ok to Reverse Lines of files up to 30 MB in size.

Operating System Limitations

The maximum file size on a FAT32 formatted partition is 4 GB or 4,294,967,296 bytes minus 2 bytes.

The maximum file size on a FAT16 partition under Windows 9.x and ME is 2 GB minus 1 byte.

But the size limit using the FAT16 file system under Windows NT or 2K is 4 GB.

And the NTFS file system has a (theoretical) maximum file size of 16 exabytes or 18,446,744,073,709,551,616 bytes (2^64 sectors)


Defining custom patterns

 Defining custom patterns
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

All memo fields (multi-line text fields) and edit fields that support patterns also provide a menu item for Insert Custom Pattern. The patterns from this list are user-defined, and come from My Documents\DataMystic\TextPipe\custom_patterns.txt, which is created when TextPipe is first run.

The format for this file is as follows:

· Comments can be placed on lines beginning with a semi colon
· Blank lines are ignored
· A pattern is introduced by a line beginning with '#'. The remainder of this line provides the pattern name which appears in the menu
· All lines after the pattern name, up to a line starting with ###END###, are treated as part of a pattern.

Here is an example custom_patterns.txt file:

;User-Defined Patterns

;

;custom_patterns.txt - saved in TextPipe's folder (the same folder

;as where textpipe.exe resides)

;

;These patterns can be inserted from the right-click or context menu on

;any TextPipe field that supports patterns

#Internet URls

(FTP|HTTP)://([_a-z\d\-]+(\.[_a-z\d\-]+)+)(([_a-z\d\-\\\./]+[_a-z\d\-\\/])+)*

###END###

#Email addresses

\b([_a-z0-9-]+(\.[_a-z0-9-]+)*)@([_a-z0-9-]+(\.[_a-z0-9-]+)*)\.([a-z]{2,3})\b

###END###

#IP addresses

\b([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\.([01]?\d\d?|2[0-4]\d|25[0-5])\b

###END###

#Credit card numbers

(\d{3,4})[- ]?(\d{4})[- ]?(\d{4})[- ]?(\d{4})

###END###


Registry entries

 Registry entries
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This page documents the TextPipe registry settings that are required to interface to it.

HKCU\SOFTWARE\DataMystic\TextPipe

showTips

0=disable tips, 1=show tips. Set this value to 0 to prevent the Tip of the Day window displaying.

showWizard

0=disable wizard, 1=show wizard. Set this value to 0 to prevent the TextPipe Wizard window displaying.

playSound

0=no sound on filter completion, 1=play sound on filter completion.

soundFile

The sound file (.wav) to play if playSound=1.

isFirstDay

0=this is not the first time TextPipe is run,1=if this is the first TextPipe is run. Set this value to 0 to prevent the help window displaying when TextPipe is run.

updateDate

Gives the next date that TextPipe should check for an update. To disable the update check, set it way in the future with 50000 (i.e. after 2017).

HKLM\SOFTWARE\DataMystic\TextPipe

rootDir

This key tells you where TextPipe is installed. TextPipe has not been installed if this key does not exist. The value of this key is TextPipe's installation folder. It is NOT guaranteed to end in a backslash, so you may have to add one manually.

TextPipeType

TextPipe comes in a number of different versions. Use this value to detect which version of TextPipe is installed and only display appropriate options. Currently these settings are only used by askSam.

TextPipeType | Internal reference | Functionality
1 | appTextPipePro | Interface enabled with editing
2 | appTextPipeStandard | Interface disabled
14 | appTextPipeProRunTime | Interface enabled without editing
18 | appTextPipeWeb | Interface disabled
20 | appTextPipeAskSam | Interface enabled with editing
21 | appTextPipeRunTimeAskSam | Interface enabled without editing

Name

Organization

Receipt

The name, organization and receipt number of this customer.


Optimizing performance

 Optimizing performance
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Memory

If you're sorting large files, give TextPipe as much memory as possible. Close EVERY unnecessary application.

Once TextPipe starts sorting, try not to start any new programs because TextPipe 'memory full' benchmark will be incorrect. TextPipe assumes you're going to give it as much memory as possible, and that it won't decrease, while it is performing a sort.

Disk I/O and virus scanners

The slowest operation that TextPipe performs is reading from and writing to disk. You can improve performance by making sure that all files being processed are stored on local disks rather than on network servers. You can also increase speed by an order of magnitude by using RAM drives - a disk held in memory, although naturally this won't help if the files you process are very large.

Disable any virus scanning while TextPipe is running.

TextPipe utilizes specific Windows API calls to enhance the speed of reading data files.

Temporary files

TextPipe doesn't use temporary files at all, except for sorting where they are unavoidable. TextPipe only ever writes out the completed output file so far. It uses a file name like TXPxxx.tmp until the file is completely written out, then it renames it to the actual output filename.

If you have enough memory, the entire sort is performed in memory for speed. Every 10000 lines, TextPipe checks to see if there is less than 16MB of physical memory (not virtual memory!). If so, it writes the sorted results so far to a temporary file and then continues. If the Output Filter is set to File Output or Single File Output, any temporary files are written to the same folder as the output file. If the Output Filter is set to Clipboard Output then any temporary files are written to the current folder. All temporary files are removed as soon as possible as the sort progresses.

Pattern matching

You can improve matching performance by an order of magnitude by allowing patterns to fail earlier by limiting what wildcards like .* can match. If you can replace .* (match any character 0 or more times) with [^>]* (match any character except '>' 0 or more times) or [^>]{0,200} (match any character except '>' up to 200 times) then your patterns will match/fail to match far more quickly.

Greedy matching is almost always slower than non-greedy matching, and if the maximum allowed match size is very large then performance will slow to a crawl.

You should avoid patterns of the type

([^">])+

where the brackets are unnecessary. Each set of brackets causes a recursive step. This pattern should be written as:

[^">]+

When the match can occur a large number of times, it can often be rewritten as

[^">]{1,10} or ([^">]){1,10}

with no change to the pattern matching process. This avoids too many recursive steps.

Also, use

Use Your Head!

See how many problems you can spot with this filter:

|--Restrict to each line in turn

| |

| |--Convert End of Lines - Auto to DOS

| | [X] Remove bad EOL

| |

| |--Replace [|] with []

| | [ ] Match case

| | [ ] Whole words only

| | [ ] Case sensitive replace

| | [ ] Prompt on replace

| | [ ] Skip prompt if identical

| | [ ] First only

| | [ ] Extract matches

| |

| |--Replace [)] with []

| | [ ] Match case

| | [ ] Whole words only

| | [ ] Case sensitive replace

| | [ ] Prompt on replace

| | [ ] Skip prompt if identical

| | [ ] First only

| | [ ] Extract matches

| |

| +--Replace [)] with []

| [ ] Match case

| [ ] Whole words only

| [ ] Case sensitive replace

| [ ] Prompt on replace

| [ ] Skip prompt if identical

| [ ] First only

| [ ] Extract matches

|

|--Restrict to each line in turn

| |

| +--Insert column 0 [@77777777]

|

|--Restrict to each line in turn

| |

| +--Insert column 0 [@88888]

|

|--Restrict to each line in turn

| |

| +--Insert column 0 [@99999]

|

|--Replace [@] with [|]

| [ ] Match case

| [ ] Whole words only

| [ ] Case sensitive replace

| [ ] Prompt on replace

| [ ] Skip prompt if identical

| [ ] First only

| [ ] Extract matches

<-- This restriction filter is unnecessary, all the filters inside it already effectively restricted to each line

1. Since the replacement text for Replacement #1, #2 and #3 is the same (blank), we could combine all three and use a

perl pattern searching for:  [()|]    OR

an EasyPattern searching for:  [ '(' or ')' or '|' ]

2.

3.

<-- This restriction filter is unnecessary, the insert column filter does not need it

1. Could be combined with #2 and #3 below, to insert all columns at the same time

<-- This restriction filter is unnecessary, the insert column filter does not need it

2. See #1

<-- This restriction filter is unnecessary, the insert column filter does not need it

3. See #1

<-- Could have been done in #1, #2 and #3 above, removing the need for a search/replace filter.

Scripting

Use of the scripting filter will slow TextPipe down because of the extra COM overhead of passing data to and from the script, and the script code being interpreted rather than compiled. Ensure that the code you write is as efficient as possible. You may be able to avoid the use of a script by judicious use of patterns, but this may be more complicated to understand. Use comments liberally - they do not slow processing down at all.

TextPipe Engine

If performance is key, you can take advantage of our embeddable TextPipe Engine. While the filter setup is done via scripting, the actual filtering is done with no GUI feedback, and no error logging for maximum performance.

Professional Services

Our consulting team offer filter optimization as a service. Please see our website for details.


Null characters - clipboard limitations

 Null characters - clipboard limitations
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

Clipboard Limitations

The Windows clipboard cannot be used to process text that contains null characters (ASCII code 0). This is because the clipboard contents are defined as null terminated, and operations on it may halt prematurely. Note that this is a Windows limitation – not a TextPipe limitation. Binary data should be place in a file for TextPipe to process it.

Regular expression limitations

While TextPipe's Perl pattern matcher DOES allow nulls to be search for, TextPipe's egrep implementation does not allow null characters to be searched for.


ASCII chart

 ASCII chart
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

ASCII = American National Standard Code for Information Interchange

Quoted-Printable encoding uses 3 digit decimal code prefixed by \ eg. CR is \013, or hex \x0a

See also: Decimal-Hex Chart

Decimal | Hex | Code | ASCII
0 | 00 | NUL | Null
1 | 01 | SOH | Start of Heading (CC)
2 | 02 | STX | Start of Text (CC)
3 | 03 | ETX | End of Text (CC)
4 | 04 | EOT | End of Transmission (CC)
5 | 05 | ENQ | Enquiry (CC)
6 | 06 | ACK | Acknowledge (CC)
7 | 07 | BEL | Bell
8 | 08 | BS | Backspace (FE)
9 | 09 | HT | Horizontal Tabulation (FE)
10 | 0a | LF | Line Feed (FE)
11 | 0b | VT | Vertical Tabulation (FE)
12 | 0c | FF | Form Feed (FE) or Page Eject
13 | 0d | CR | Carriage Return (FE)
14 | 0e | SO | Shift Out
15 | 0f | SI | Shift In
16 | 10 | DLE | Data Link Escape (CC)
17 | 11 | DC1 | Device Control 1
18 | 12 | DC2 | Device Control 2
19 | 13 | DC3 | Device Control 3
20 | 14 | DC4 | Device Control 4
21 | 15 | NAK | Negative Acknowledge (CC)
22 | 16 | SYN | Synchronous Idle (CC)
23 | 17 | ETB | End of Transmission Block (CC)
24 | 18 | CAN | Cancel
25 | 19 | EM | End of Medium
26 | 1a | SUB or EOF | Substitute or End of File
27 | 1b | ESC | Escape
28 | 1c | FS | File Separator (IS)
29 | 1d | GS | Group Separator (IS)
30 | 1e | RS | Record Separator (IS)
31 | 1f | US | Unit Separator (IS)
32 | 20 | SP | Space
33 | 21 | ! | Exclamation Point
34 | 22 | " | Quotation Mark
35 | 23 | # | Number Sign, Octothorp, "pound"
36 | 24 | $ | Dollar Sign
37 | 25 | % | Percent
38 | 26 | & | Ampersand
39 | 27 | ' | Apostrophe, Prime
40 | 28 | ( | Left Parenthesis
41 | 29 | ) | Right Parenthesis
42 | 2a | * | Asterisk, "star"
43 | 2b | + | Plus Sign
44 | 2c | , | Comma
45 | 2d | - | Hyphen, Minus Sign
46 | 2e | . | Period, Decimal Point, "dot"
47 | 2f | / | Slash, Virgule
48 | 30 | 0 | 0
49 | 31 | 1 | 1
50 | 32 | 2 | 2
51 | 33 | 3 | 3
52 | 34 | 4 | 4
53 | 35 | 5 | 5
54 | 36 | 6 | 6
55 | 37 | 7 | 7
56 | 38 | 8 | 8
57 | 39 | 9 | 9
58 | 3a | : | Colon
59 | 3b | ; | Semicolon
60 | 3c | < | Less-than Sign
61 | 3d | = | Equal Sign
62 | 3e | > | Greater-than Sign
63 | 3f | ? | Question Mark
64 | 40 | @ | At Sign
65 | 41 | A | A
66 | 42 | B | B
67 | 43 | C | C
68 | 44 | D | D
69 | 45 | E | E
70 | 46 | F | F
71 | 47 | G | G
72 | 48 | H | H
73 | 49 | I | I
74 | 4a | J | J
75 | 4b | K | K
76 | 4c | L | L
77 | 4d | M | M
78 | 4e | N | N
79 | 4f | O | O
80 | 50 | P | P
81 | 51 | Q | Q
82 | 52 | R | R
83 | 53 | S | S
84 | 54 | T | T
85 | 55 | U | U
86 | 56 | V | V
87 | 57 | W | W
88 | 58 | X | X
89 | 59 | Y | Y
90 | 5a | Z | Z
91 | 5b | [ | Opening Bracket
92 | 5c | \ | Reverse Slant
93 | 5d | ] | Closing Bracket
94 | 5e | ^ | Circumflex, Caret
95 | 5f | _ | Underline, Underscore
96 | 60 | ` | Grave Accent
97 | 61 | a | a
98 | 62 | b | b
99 | 63 | c | c
100 | 64 | d | d
101 | 65 | e | e
102 | 66 | f | f
103 | 67 | g | g
104 | 68 | h | h
105 | 69 | i | i
106 | 6a | j | j
107 | 6b | k | k
108 | 6c | l | l
109 | 6d | m | m
110 | 6e | n | n
111 | 6f | o | o
112 | 70 | p | p
113 | 71 | q | q
114 | 72 | r | r
115 | 73 | s | s
116 | 74 | t | t
117 | 75 | u | u
118 | 76 | v | v
119 | 77 | w | w
120 | 78 | x | x
121 | 79 | y | y
122 | 7a | z | z
123 | 7b | { | Opening Brace
124 | 7c | | | Vertical Line
125 | 7d | } | Closing Brace
126 | 7e | ~ | Tilde
127 | 7f | DEL | Delete
128 | 80 |  | Reserved
129 | 81 |  | Reserved
130 | 82 |  | Reserved
131 | 83 |  | Reserved
132 | 84 | IND | Index (FE)
133 | 85 | NEL | Next Line (FE)
134 | 86 | SSA | Start of Selected Area
135 | 87 | ESA | End of Selected Area
136 | 88 | HTS | Horizontal Tabulation Set (FE)
137 | 89 | HTJ | Horizontal Tabulation with Justification (FE)
138 | 8a | VTS | Vertical Tabulation Set (FE)
139 | 8b | PLD | Partial Line Down (FE)
140 | 8c | PLU | Partial Line Up (FE)
141 | 8d | RI | Reverse Index (FE)
142 | 8e | SS2 | Single Shift Two (1)
143 | 8f | SS3 | Single Shift Three (1)
144 | 90 | DCS | Device Control String (2)
145 | 91 | PU1 | Private Use One
146 | 92 | PU2 | Private Use Two
147 | 93 | STS | Set Transmit State
148 | 94 | CCH | Cancel Character
149 | 95 | MW | Message Waiting
150 | 96 | SPA | Start of Protected Area
151 | 97 | EPA | End of Protected Area
152 | 98 |  | Reserved
153 | 99 |  | Reserved
154 | 9a |  | Reserved
155 | 9b | CSI | Control Sequence Introducer (1)
156 | 9c | ST | String Terminator (2)
157 | 9d | OSC | Operating System Command (2)
158 | 9e | PM | Privacy Message (2)
159 | 9f | APC | Application Program Command (2)
160-254 | a0-fe |  |
255 | ff |  | Invisible space

NOTES:

(1) Introducer

(2) String Delimiter

(CC) Communication Control

(FE) Format effector

(IS) Information Separator

Unprintable Characters


Decimal to Hex Conversion Chart

 Decimal to Hex Conversion Chart
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

See also: ASCII chart

 | Second hex digit (least significant, or rightermost)
First hex digit | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | A | B | C | D | E | F
0 | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15
1 | 16 | 17 | 18 | 19 | 20 | 21 | 22 | 23 | 24 | 25 | 26 | 27 | 28 | 29 | 30 | 31
2 | 32 | 33 | 34 | 35 | 36 | 37 | 38 | 39 | 40 | 41 | 42 | 43 | 44 | 45 | 46 | 47
3 | 48 | 49 | 50 | 51 | 52 | 53 | 54 | 55 | 56 | 57 | 58 | 59 | 60 | 61 | 62 | 63
4 | 64 | 65 | 66 | 67 | 68 | 69 | 70 | 71 | 72 | 73 | 74 | 75 | 76 | 77 | 78 | 79
5 | 80 | 81 | 82 | 83 | 84 | 85 | 86 | 87 | 88 | 89 | 90 | 91 | 92 | 93 | 94 | 95
6 | 96 | 97 | 98 | 99 | 100 | 101 | 102 | 103 | 104 | 105 | 106 | 107 | 108 | 109 | 110 | 111
7 | 112 | 113 | 114 | 115 | 116 | 117 | 118 | 119 | 120 | 121 | 122 | 123 | 124 | 125 | 126 | 127
8 | 128 | 129 | 130 | 131 | 132 | 133 | 134 | 135 | 136 | 137 | 138 | 139 | 140 | 141 | 142 | 143
9 | 144 | 145 | 146 | 147 | 148 | 149 | 150 | 151 | 152 | 153 | 154 | 155 | 156 | 157 | 158 | 159
A | 160 | 161 | 162 | 163 | 164 | 165 | 166 | 167 | 168 | 169 | 170 | 171 | 172 | 173 | 174 | 175
B | 176 | 177 | 178 | 179 | 180 | 181 | 182 | 183 | 184 | 185 | 186 | 187 | 188 | 189 | 190 | 191
C | 192 | 193 | 194 | 195 | 196 | 197 | 198 | 199 | 200 | 201 | 202 | 203 | 204 | 205 | 206 | 207
D | 208 | 209 | 210 | 211 | 212 | 213 | 214 | 215 | 216 | 217 | 218 | 219 | 220 | 221 | 222 | 223
E | 224 | 225 | 226 | 227 | 228 | 229 | 230 | 231 | 232 | 233 | 234 | 235 | 236 | 237 | 238 | 239
F | 240 | 241 | 242 | 243 | 244 | 245 | 246 | 247 | 248 | 249 | 250 | 251 | 252 | 253 | 254 | 255


Unprintable Characters

 Unprintable Characters
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

TextPipe allows you to embed control characters in search and replace strings, line prefixes and user-defined replacement strings. All fields that support entering unprintable characters in this manner have a "context" or right-click menu that allows them to be entered without remembering the exact code.

E.g. To search for "hello" followed by a tab followed by "there" followed by a DOS end of line, enter the search string as

hello\tthere\r\n

where \t is the code for a Tab (see Standard ASCII codes).

Other Special Codes

Code | ASCII value | Description
\a | 7 | Bell
\b | 8 | Backspace
\cC |  | Control character. ie \cA inserts a Ctrl+A, \cM inserts a Ctrl+M
\t | 9 | Horizontal tab
\n | 10 | Line feed
\v | 11 | Vertical tab. You can also use \x0b, perl patterns do NOT support \v
\f | 12 | Form feed
\r | 13 | Carriage return
\e | 27 | Escape
\xXX | Any | Any character code, specified in hex. The XX is replaced by the character's 2 digit hex code
\x{ .. } | Any | Any sequence of character codes, specified in hex. This can be very handy for specifying unicode code sequences, such as \x00000020 for a unicode space character, instead of \x00\x00\x00\x20. To enter a two-byte number, use \x{0020}. For 3 bytes, use \x{000020}. For 4 bytes use \x{00000020}.

See also

Macros


Filters Menu

 Filters Menu
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

The filters menu contains over 100 different ways to manipulate your data.

· Convert
· Add
· Remove
· Unicode
· Replace
· Special
· Map
· Email
· Restrict

Filter wizard

The Filter Wizard provides a step-by-step method for beginners to construct a TextPipe filter list.

Web-based filter wizard (task based)

An alternate approach, this filter wizard shows how easy it is to use a web page to control TextPipe to perform custom tasks.


Restrict XML-delimited field

 Restrict XML-delimited field
Menus: File Edit Tools Window Help
Filters: Wizards Convert Unicode Add Remove Replace Extract Special Maps Restrict

This filter selects an XML-delimited field for further processing, such as a search/replace inside it, or a case conversion etc.

For example, if your data looked like this, and you wanted to UPPERCASE the CompanyName field:

<TextPipeRow>

<CustomerID>ALFKI</CustomerID>

<CompanyName>Alfreds Futterkiste</CompanyName>

<ContactName>Maria Anders</ContactName>

<ContactTitle>Sales Representative</ContactTitle>

<Address>Obere Str. 57</Address>

<City>Berlin</City>

<Region></Region>

<PostalCode>12209</PostalCode>

<Country>Germany</Country>

<Phone>030-0074321</Phone>

<Fax>030-0076545</Fax>

</TextPipeRow>

Specify the 'Restrict to tag' name to select as 'CompanyName'. Then use a sub filter of UPPERCASE. The resulting output looks like this:

<TextPipeRow>

<CustomerID>ALFKI</CustomerID>

<COMPANYNAME>ALFREDS FUTTERKISTE</COMPANYNAME>

<ContactName>Maria Anders</ContactName>

<ContactTitle>Sales Representative</ContactTitle>

<Address>Obere Str. 57</Address>

<City>Berlin</City>

<Region></Region>

<PostalCode>12209</PostalCode>

<Country>Germany</Country>

<Phone>030-0074321</Phone>

<Fax>030-0076545</Fax>

</TextPipeRow>

Note - if you want to change text inside the XML tag itself, use a Find text inside an HTML tag filter.

See also

Sub filters

Restrict to delimited fields (CSV, Tab, Pipe etc)


What's New in TextPipe v9.7.3 – August 22, 2014
===============================================

*
Removed warning about active COM objects.

What's New in TextPipe v9.7.2 – August 15, 2014
===============================================

*
Fixed bug in Remove lines matching perl pattern.

What's New in TextPipe v9.7.1 – August 14, 2014
===============================================

*
Difference tool integration - new function to open a file in text difference
tool – in Trial Run, from the Files Tab, and from the Status Results Window.
Integrate with tools such as WinDiff and BeyondCompare.
*
Sort function now much faster.
*
Removed locking of TextPipe into the COM Running Object table, so COM errors
do not occur on completion of scripts.
*
Upgraded PDF component.
*
Display text now outputs 'simultaneous search' and 'Process longest strings
first' options.
*
Fixed bug in Convert Excel to Text filter – was ignoring non-hidden sheets.

What's New in TextPipe v9.7 – May 29, 2014
==========================================

*
Mainframe copybook filter now allows pass-thru EBCDIC data ie BINARY data
e.g. using PIC X(48) USAGE IS BINARY.
Mainframe copybook filter now does not insert a null (\x00) character after
fields in Fixed Width mode.
Invalid copybooks now send errors to log when running from command line.
*
New EasyPatterns:
    [Drive]    e.g. C:   E: -> returns drive letter in @drive@
    [Folder]   e.g. \My Documents\  -> returns 'My Documents' in @folder@
    [Path]     e.g. d:\My Documents\  -> returns 'd:\My Documents' in
                    @drive@ and @path@
    [UNCPath]  e.g. \\server\share\path\filename -> returns @server@,
                    @share@ and @path
    [Filename] e.g. testdocument.doc -> returns @filename@
*
Unicode conversion filter did not retain the Error Character if the Input or
Output encodings were not changed.
*
Hidden worksheets are now ignored by the Excel to Text filter.
*
Split filter now cleans up its temporary files.

What's New in TextPipe v9.6 – January 1, 2014
=============================================

*
Trial run area now shows output progressively – giving important feedback
earlier in the filter development lifecycle.
*
Now shows '(more pending)' next to the file count if more files remain to be
counted, so users aren't misled that only 1000 files will be processed.
*
Added new filter list menu items for Remove exactly, Remove perl pattern, and
Remove EasyPattern
*
File grid now allows mass-changing of Action or Subfolder columns using
right-click menu after selecting rows in the row number column.
*
File grid ensures that the top blank row is used with drag/drop.
*
Pressing [Tab] in a memo control now inserts a tab – it no longer changes
focus to the next field.
*
Updated Perl regexp third-party package.
*
Updated PDF third-party package.
*
Re-ordered scratch pad and gave tabs consecutive numbers.
*
'Delete' action of search/replace grid now much faster for entire-grid
clearing.
*
TextPipe now warns if a Convert Word/Excel to Text filter is used at the same
time as the Input Filter/Process Inside Compressed files option – these are
not compatible.
*
Fixed missing comma in VBScript/Jscript code generation for search/replace
lists.
*
Installer - fixed x32 shell extensions missing under Windows x32.
*
Replace List path is now relative to the filter path.
*
Using File Menu\Open to open non-filter files, now adds them to the current
Files To Process list.

What's New in TextPipe v9.5.3 – August 5, 2013
==============================================

*
Fixed error with Restrict to Blocks of Length X and several other filters
causing out of memory errors, as they were confused for 'repeat filter X
times' filters.

What's New in TextPipe v9.5.2 – July 25, 2013
=============================================

*
Fixed 'Only log when a replace occurs' not working for consecutive search
replace filters in the Search/Replace list filter.
*
Fixed UTF-8 mode perl pattern replace strings from being converted to utf-8.

What's New in TextPipe v9.5.1 – July 15, 2013
=============================================

*
Added Help entry for Repeat Subfilters X Times.
*
Adding filters with no valid filter window now creates a new filter window.
*
When Utf8 Support is checked for replace filters, Unicode characters in the
search for and replace with fields are now correctly converted to UTF-8 prior
to being processed by the PCRE engine.
*
Clicking [Delete] on empty areas of the Files grid no longer causes errors.

What's New in TextPipe v9.5 – July 3, 2013
==========================================

*
Added log entries for size/date of ignored files.
*
Search/replace lists can now generate log entries – useful for debugging.
Also, logs can be output only for where replacements occurred, which keeps
the log size manageable when importing large search/replace lists (default On
when loading existing filters as previously no logging was done).
*
EasyPatterns v2.8 – new keywords EBCDICletter, EBCDICupper, EBCDIClower,
DayNameShort, DayNameLong, MonthNameShort, MonthNameLong, YearShort,
YearLong, Float, Integer. WordBreak is a new synonym for WordBoundary. The
behaviour of number, numbers and numeric has been changed so that instead of
matching a single digit (which is already handled by the digit keyword), they
now match a number with an optional sign and optional decimal point.
*
Mainframe copybook filter now copes with VALUES of ZERO/ZEROS/ZEROES,
SPACE/SPACES and multiple comma-separated values in addition to the string
and integer values previously supported. Now supports VALUE clause on numeric
fields.
*
New right-click menu items in all filters view, to Link To or Insert filter
files.
*
Filter comments can now be included in the output log, from the filter
options settings at the top of the tree. This greatly assists in debugging
complex filters.
*
New filter to Repeat Subfilters X Times; useful when you are not sure how
many times a search/replace pattern needs to be applied.
*
Named subexpressions created by a Split on Pattern filter are now saved as
global variables for use in other filters. Useful for pushing split pattern
text into filenames.
*
If Prompt On Replace is disabled during a filter list, the filter is now
marked as modified when the Prompt form is closed.
*
Search/replace lists now discard blank search terms and terms where the
replacement is identical to the search.
*
Log filename now has environment variables resolved before display.
*
Partial trial runs now reset the status window properly.
*
Better handling of split filenames with format string errors in them.
*
Filter components relative to the filter path now work in Unicode folder
names.
*
Restrict to Inside filenames is now available in the Lite version to assist
batch file re-naming.
*
Upgraded PDF component from 8.15 to 9.13 – now handles a greater range of PDF
document types.
*
Updated to Zip Master 1.9.1 and PCRE v7.5.
*
Fixed handling of EasyPattern Ascii() keyword.
*
Fixed bug with Right-click\Delete on files in the file list.
*
Fixed bug in search/replace list, in perl pattern match mode. If the search
term contained brackets (), then replacement phrases on rows following it
were mismatched.
*
Split filter now copes with absolute pathnames in the split filename. Split
filter now outputs log entries when it starts a new file, and outputs
separate logging details for each split file.
*
Number of files processed now does not include files inside zip containers.
The current file being processed now shows the zip container and the internal
filename.
*
Fixed export of Search/Replace lists filters to VBScript.
*
Fixed incorrect path for shell extensions on install.
*
Fixed bug with handling of moving a delimited field, where the line did not
end with the expected delimiter.

What's New in TextPipe v9.4.1a – April 18, 2013
===============================================

*
Bug fix for Retain modified dates for the compressed file.
*
Matrix search now uses slightly faster initialization method.

What's New in TextPipe v9.4.1 – April 9, 2013
=============================================

*
Fixed installer issue on x32 systems.
*
Retain modified dates now works for compressed files (for the compressed file
itself).

What's New in TextPipe v9.4 – April 3, 2013
===========================================

*
Matrix (parallel) search/replace for Exact replacements:
- now supports First Only option – any term that has already been replaced
will not be replaced again.
- now supports Case Sensitive replace- where the case of the replacement is
based on the case of the found text.
- now supports prompt on replace.
- now much faster - 600,000 replacements against 8MB compressed .xlsx file in
31 seconds.
- fixed bug where changes were not counted and hence the output file was
never updated unless changes had been made by other filters.
- fixed bug where text that did not match was output in lowercase when Match
Case was unchecked.
- fixed bug where text at the end of the file or restriction was not
replaced.
*
Added .XPS as a compressed document type.
*
Word to Text Filter and PDF to Text Filter now set output filename to .txt,
Excel to CSV filter now sets output filename to .csv.
*
New warnings to ensure Process Binary files is checked when Process Inside
Compressed Files is checked.
*
TextPipe now switches immediately to the Status tab when a trial run is
started so that progress can be reviewed, and then switches back to the Trial
Run tab on completion.
*
Exact search replace now does not even allow special characters.
*
New status messages from search/replace list filters to give progress updates
when loading large files.
*
New filter to restrict processing to specified files within a zip or other
compressed file.
*
Compressed file handler now removes any left over files at the end of the
filter job.
*
Compressed file handler now shows status information as it extracts and re-
compresses files.
*
New status logs for filters that perform end-of-line conversions.
*
New validation check for 'Send variable N to subfilter' to ensure there are
at least N captured expressions.
*
Now handles any PKZIP compressed file types regardless of whether the
extension is known or not.
*
Fixed bug with handling compressed files where modified files were not
updated back into the original compressed file.
*
Fixed bug where AV occurred when adding a new file to a file list that had
been cleared using [Clear All].
*
Fixed export to clipboard of search/replace lists – the second and subsequent
lines were not indented correctly.
*
Removed 'Warning: Find and Replace string lengths don't match' message when
the Action is not set to 'Replace'.

What's New in TextPipe v9.3.1 – January 8, 2013
===============================================

*
Fixed problem with endless loop caused when Up or Down arrow pressed in Files
Grid.

What's New in TextPipe v9.3 – December 18, 2012
===============================================

*
File grid now supports Unicode characters.
*
Support for Unicode pattern matches in the files grid (perl regex mode).
*
Split filter now allows Unicode filenames, and Unicode file break patterns.
*
Grep filters now allow Unicode patterns (when UTF-8 support mode is enabled).

What's New in TextPipe v9.2.2 – September 6, 2012
=================================================

*
Fixed installer error with registry entries required by x64 Shell Extension

What's New in TextPipe v9.2.1 – September 5, 2012
=================================================

*
Fixed crash with sorting more than one file.
*
Fixed crash experienced by some users caused by the x64 shell extension when
the desktop was right-clicked.

What's New in TextPipe v9.2 – August 12, 2012
=============================================

*
Massive performance improvement for large added strings, such as headers and
footers.
*
Mainframe copybook filter now handles the option 'Process numeric fields
only' properly – previously it output blanks for alpha fields in this mode.
*
New popup menu for the status log – options to copy the log entry, copy all
log entries, open the referenced file or its containing folder.
*
New filter Convert HTML/XML Entities to Text, as it handles both numeric and
253 named entities.
*
New 'Copy to Clipboard' button for Search/Replace list filter.
*
Right-click menus now display available environment variables and their
current values, for easy insertion.
*
Updated internal PCRE (Pattern Matching) engine to v8.30 and support for
Unicode 6.1.0.
*
The perl regex position '^' for the start of the file now works with Unix
files.
*
TextPipe Engine now returns an error if the output file is open in another
application – in the same way the GUI does.
*
Improved COM object model to have less embedded references to objects, making
it easier to free the objects. Also, calling the output* and input* functions
when calling the TextPipe.FilterWindow object no longer result in errors.
*
Uninstall from now on will not remove settings.
*
'updateDate' in registry can now be set to zero to disable update checking.
*
TextPipe Engine now ignores Ctrl+C and Ctrl+Break.
*
MIME Base 64 Decoder now excludes the trailing garbage character.
*
Output Filter now allows macros and ENV vars in the single output filename.
*
Fixed threading errors when cancelling running jobs.
*
Debug window and embedded help windows now record their size correctly.
*
Stack overflow, as in the case of pattern matches with poor constraints, are
now trapped as Exit Code 254.
*
Fixed environment variable substitution error in filenames.

What's New in TextPipe v9.1 – March 15, 2012
============================================

*
Added new macro @inputPath to contain the path to the input file, with a
trailing slash. This lets you easily create output folders at the input path
location.
*
Added new macro @inputExtension to contain the extension of the current input
file.
*
Output Filter's Single Output Filename, new File Extension and New Path can
now all contain macros.
*
Added support for Open Office Writer (.ODT), Calc (.ODS) and Impress (.ODP)
file types as new compressed file types.
*
HTML Restriction filter now supports entity names with embedded colons (:)
and hyphens (-) which is essential for working with Open Office files and
other XML types.
*
EasyPattern keyword [backwardSingleQuote] renamed to [graveAccent] as the
original name was incorrect.
*
The right-click context menu on the Filter Tree now allows groups of selected
filters for 'Make Into a Subfilter' and 'Promote'.
*
Fixed operation of log 'overwrite' option – the log cache would not clear
previous logs created since TextPipe was started. Logs were also not being
closed at the completion of a filter job, hence they could not be deleted.
*
Macros and environment variables are now allowed to contain numbers, provided
that the first character is not a number.
*
The right-click context menu of the replace string now inserts captured
variable names instead of $1, $2 etc when you ask it to insert a CSV, Tab or
XML string. This makes it far easier to identify which fields belong where,
as well as making the replacement string less sensitive to changes to the
search pattern.
*
New optimization for named captured variables – TextPipe replaces these with
the captured variable number behind the scenes – so you get the ease of use
benefits of named variables, with no speed decrease.
*
EasyPatterns now allow a literal (ie a string of characters) inside a 'not'
expression, e.g. not 'ABC', or not EBCDIC('HDR').
*
Mainframe copybook filter can now remove FILLER fields inside OCCURS blocks.
*
Supported but not installed code pages now show 'NOT INSTALLED' beside them.
*
Fixed AV's when saving or loading a file when running under Windows 7.
*
Sort filter set to maximum allowed memory of 2GB to avoid integer overflows.

What's New in TextPipe v9.0 – January 30, 2012
==============================================

*
New filter to perform a calculation on restricted text e.g. add 1.5 to a
field.
*
Comment filter now allows Unicode text.
*
Windows Home Server no longer considered a 'Server' platform.
*
Log files now support Unicode.
*
Fixed action of 'Skip/Replace in Rest of File' when used inside a restriction
– previously it carried forward for the rest of the job, instead of the rest
of the current file.
*
Remove multiple whitespace filter now generates detailed log information.
*
Trial Runs now generate the same logging data as real runs.
*
Fixed missing filters from Lite Edition.
*
TextPipe only registers its COM object during install and for the /REGSERVER
command line option – preventing errors on less-than-admin use.
*
Restored action of F9 button to start running when the All Filters view is
selected.
*
Opening and Updating zip file log messages are now clickable.
*
Updated third-party zip DLL to improve Windows x64 support.
*
Fixed handling of double backslashes in the replacement term for
search/replace lists when consecutive search was selected.
*
Unicode strings are now converted to UTF8 when the perl pattern filter has
UTF8 mode selected.
*
Restrict to matching/non-matching filenames now works with files of unlimited
size.

What's New in TextPipe v8.9.9 – November 10, 2011
=================================================

*
Fixed Sort filter allowing duplicates.
*
Entering \x{1234} when UTF-8 mode is enabled for a perl pattern match now
does not show a validation message.
*
Updated PCRE engine.
*
Fixed scrolling of found text into view in Prompt On Replace dialog.
*
New [Add Files] button on Files to Process tab.
*
Combo boxes now work with right-click context menus.
*
Search/replace list import now supports Unicode.

What's New in TextPipe v8.9.8.1 – October 24, 2011
==================================================

*
Updated PDF libraries for extraction from more recent file formats.
*
Added large [Add Files] button to Files to Process tab.

What's New in TextPipe v8.9.8 – October 24, 2011
================================================

*
Capture Text, Break on Value Change window now shows length of strings and
current cursor position.
*
Updated internal PCRE (Pattern Matching ) engine to v8.13 and support for
Unicode 6.0.0.
*
Updated Unicode internal libraries to support Unicode 4.1 for Normalization
etc.
*
COM callees are now notified of Stack violations (e.g. during pattern
execution) or other critical errors via the existing 'FilterWindow.errorText'
variable.
*
Only modified files are added back into Zip files such as .zip, .docx, .xlsx
and .pptx.

What's New in TextPipe v8.9.7 – October 13, 2011
================================================

*
Updated x64 extension to only show current file name, not full path.
*
Better checking for changed files with Extract matching/non-matching lines
and Remove matching/non-matching lines.
*
All Available Filters tree now show properly when the text 'Search...' is
translated.
*
Fixed errors with TP Engine not unloading cleanly, and not returning data.

What's New in TextPipe v8.9.6 – September 15, 2011
==================================================

*
Windows Explorer x64 extension working!

What's New in TextPipe v8.9.5 – September 2, 2011
=================================================

*
Fixed dimming of unavailable filters in Lite and Standard versions.

What's New in TextPipe v8.9.4 – August 18, 2011
===============================================

*
Fixed Browse buttons not resizing correctly.
*
Converted file list field to Unicode.
*
New 'Close' buttons in basic forms (editor, copybook tree viewer).
*
Now handles 'Convert Word/Excel to text' inside subfilters, prior to the job
starting (e.g. for processing exclusion lists etc).
*
Fixed Mainframe copybook filter handling of OCCURS .. DEPENDING ON .. when
the Trim Fields option was checked.
*
Fixed display of Unicode text in various multi-line edit fields.

What's New in TextPipe v8.9.3 – July 26, 2011
=============================================

*
Fixed crash on filter run when working with files.
*
New 'Open Containing Folder' for filter lists.
*
Fixed bug when double-clicking on filenames.
*
Filter search now works for filters like 'Sort' – where previously the
invisible short cut character & prevented this from working.
*
Search/replace lists now returned to naïve UTF-8 handling.
*
Binary test now works for Unicode filenames.

What's New in TextPipe v8.9.2 – July 19, 2011
=============================================

*
Fixed bug with the way that PlaySound was called at the end of the filter job
which caused crashes.

What's New in TextPipe v8.9.1 – July 8, 2011
============================================

*
Removed windows explorer x64 extension as buggy.
*
Search/replace lists now check for Unicode external files.

What's New in TextPipe v8.9 – July 8, 2011
==========================================

*
New search function to quickly find matching filters or external files.
*
Simultaneous search now works with patterns – not just Exact match.
*
Filter move Up/Down buttons in short cut menu.
*
Enable/Disable prompting in short cut menu to easily debug large groups of
filters.
*
Secondary output filters are no longer removed when pasting.
*
Log files are now opened in their associated app rather than in Notepad when
double-clicked or when [Show] button clicked.
*
Better Unicode filename support when filenames are launched by double-
clicking on text fields.
*
Updated internal pattern matching libraries.
*
TextPipe Engine now logs just as the GUI can.
*
Unicode filenames now supported by Merge Filename; Output Extension, New Base
Folder and Single Output Filename; Split Filename; Log filename; Map comment;
String filters' filenames, Search/Replace List filename.
*
Unicode filename support in status grid, file grid, processing etc.

What's New in TextPipe v8.8.2 – April 13, 2011
==============================================

*
Dimmed filters now cannot be double-clicked to add them.
*
While loading a search/replace list, TP now responds to events.
*
Link Filter now in every edition.
*
List filename now gets loaded properly.

What's New in TextPipe v8.8.1 – April 11, 2011
==============================================

*
Added .ODT as a compressed file type so that OpenDocument documents can be
handled just like Office .DOCX format compressed XML files.
*
New Files to Process field to load files from a list file (note – this
ability was already available from the command line with /L= and from COM via
addFileList).
*
The Replace List filter now has a button to paste a list from the clipboard.
*
Added new exit code of 255 for abnormal termination due to textpipe.exe being
killed or an unexpected error.
*
Deferred hooking into Windows HTML help system for faster startup.
*
Migration of filters from Pro to Standard, and Casing filters from Standard
to Lite.

What's New in TextPipe v8.8 – March 24, 2011
============================================

*
New file date and file size filters to help select files to be processed.
*
New context menu items for filter list, to duplicate, demote or promote
filters.
*
Renamed variables in Extract-generated .vbs and .js scripts for consistency
and readability.
*
Lite and Standard version now show dimmed menu items for unavailable options.
*
Upgraded PCRE pattern engine.
*
Improved speed of double-quoting and trimming mechanism in Mainframe Copybook
filter.
*
New EasyPatterns [DayNameLocal] and [MonthNameLocal].
*
Default for loading older Mainframe Copybook filters without Force Quotes
options is now set to unchecked.
*
Fixed EasyPattern keyword 'atomic'.
*
Fixed coInitialize/coUninitialize COM errors with repeated running of the
scripting filter.

What's New in TextPipe v8.7 – February 14, 2011
===============================================

*
Comment filters and other filters with multi-line values now display just the
first non-blank line in My Filter List.
*
Various GUI fixes.
*
New macro @outputFilename to output the name of the output file without the
path.
*
Correct detection of Windows XP x64 as a non-server O/S.
*
Updated code to detect My Documents and AppsData folder under Windows 7.
*
Convert Excel to Text filter now outputs a form feed (ASC 12) character
between worksheets.
*
Filter lists are now freed on completion. E.g. Filters that load large
suppression lists (e.g. 300MB Remove matching lines filters) can now be run
repeatedly without restarts.
*
Better error trapping for Out of Memory errors and other exceptions that
occur during filter execution.
*
Many memory leaks fixed, making working with large files multiple times far
easier.

What's New in TextPipe v8.6.8 – November 16, 2010
=================================================

*
Speed improvements for Remove Lines From List, and Retain Lines From List,
when using large external compare files (10MB+).

What's New in TextPipe v8.6.7 – October 27, 2010
================================================

*
The Output Filter setting of 'Backup Mode' now does indeed rename the
original input file to '.bak' rather than operating on the output filename.

What's New in TextPipe v8.6.6 – October 19, 2010
================================================

*
Setting the global variable @CLIPBOARD using a Capture Filter or a named
variable in a search pattern now sets the Windows clipboard as well.
*
Missing Recently Used files are now removed from the list.
*
New 'atomic' keyword allows EasyPatterns to match either all or none of a
group e.g.
  [ atomic(0+ spaces) ]
matches all the spaces that are present, or none of them – and does not
backtrack and keep retrying with one less space each time. This makes
matching far faster.
*
Mainframe Copybook Filter in the Server Edition no longer automatically adds
a Split CPU filter.

What's New in TextPipe v8.6.5 – September 8, 2010
=================================================

*
Shell extension now supports Unicode filenames – fixed error with only the
first letter of menu options being shown.

What's New in TextPipe v8.6.4 – August 25, 2010
===============================================

*
Mainframe copybook filter now allows trailing minus sign to identify signed
fields e.g. PIC 9(11)V99-.
*
Fixed problem where status messages were occasionally lost or doubled up.
*
Improved Unicode file handling.

What's New in TextPipe v8.6.3 – August 13, 2010
===============================================

*
Fixed bug with output files causing file rename errors.

What's New in TextPipe v8.6.2 – August 11, 2010
===============================================

*
Sound on completion when run from command line now uses settings from
Preferences (rather than being forced on). Can be overridden using
/PLAYSOUND.
*
Updated Explorer Context Menu to support Unicode.
*
Now uses updated zip library.
*
Partial support for Unicode filenames.
*
Fixed bug with Ignore Case always enabled, for Count Duplicate Lines and
Remove Duplicate Lines filters.
*
Fixed bug where @ and $ signs in the replace string were handled incorrectly-
when used in the Search/Replace list filter in sequential mode.

What's New in TextPipe v8.6.1b – July 2, 2010
=============================================

*
Change to update mechanism to prevent spurious update notifications.

What's New in TextPipe v8.6.1 – July 1, 2010
============================================

*
Log file is now closed and re-opened every 30 seconds so that the file size
increases over time. If you open the file it will still show the currently
written data.
*
Fixed input file gathering – new files are now available for processing
immediately rather than grouped by folder. Previously, folders with thousands
of files would cause an apparent stop in processing.
*
Uninstaller now does not require explorer.exe (Explorers and Task bar) to be
closed; reboot or logoff required as before.
*
Cancel button now does not pause thread until [Ok] clicked, due to conflicts
with suspending the thread.

What's New in TextPipe v8.6 – June 16, 2010
===========================================

*
File list is generated in parallel with processing – so work commences
immediately, even with millions of files to process.
*
Status window now shows expected completion time based on total data volume
being processed.
*
Mainframe Copybook Filter – adding quotes to alpha fields is now an option
rather than automatic. Quotes and field delimiters in the record header and
numeric fields are now escaped appropriately.
*
'Delete Input Files On Completion' Input Filter warning message now disabled
in command line and Engine mode.
*
Scripting filter now available in the Standard version.
*
Performance improvement for Switch CPU filter.
*
Restrict to matching lines filter now uses its subfilters to determine if a
change has occurred to the input file.
*
Fixed bug with Split CPU filter when used with multiple output files.

What's New in TextPipe v8.5.3 – May 19, 2010
============================================

*
Fixed bug in MultiCPU version with Split CPU filter stopping output files
being written.
*
Merge filename is now added to the COM output files list.
*
Fixed TextPipe Engine bug with Edit Distance search/replace and Convert to
Word List not working on first filter execution.
*
Fixed Insert Filter option changing filename and inserting extra Output
filter.

What's New in TextPipe v8.5.2 – May 11, 2010
============================================

*
If the Output file cannot be written to (e.g. the network resource is no
longer available), then the Input filter option to Delete Input Files is
ignored. The Open Output Files On Completion option is also ignored in this
case.
*
When run from the command line, errors (such as no matching files to process)
now do not require manual intervention. They are available in the log if it
is turned on.
*
If TextPipe cannot create a temporary file on the target drive (e.g. if it is
a network drive), then it falls back to creating it on the local machine in
%TEMP% and copies it across at the end of the job.

What's New in TextPipe v8.5.1 – April 15, 2010
==============================================

*
Fixed bug with HTML filter when 'Include text' is unchecked, the restriction
is designed to match text between open and close tags, and the open tag has
no text after the tag name e.g. <salesCommissions>...</salesCommissions>

What's New in TextPipe v8.5 – April 7, 2010
===========================================

*
Split filename can now include macros using @varname syntax.
*
Database filter now copes with multiple record sets from a single query.
*
Changed installer to not require admin, for Vista and Windows 7.
*
TextPipe now does not require admin rights to run – which in Windows
Explorer:
- restores ability to double-clicking a filter to open it
- restores the TextPipe context menu
- restores drag and drop.
*
Clipboard input and output can now both be in Unicode format.
*
New COM method terminate() to allow filters to be stopped when called from
execute() or executeClipboard().
*
TextPipe Engine now supports the Split CPU filter.
*
Fixed bug that occurred with T-Filters and Uppercase/Lowercase filters. The
Upper/Lower filters overwrote memory so that modified text could occur in the
alternate stream of the T-Filter.
*
Fixed filter 'Restrict to filenames matching pattern'.

What's New in TextPipe v8.4.9.1 – February 24, 2010
===================================================

*
Mainframe copybook filter – now displays a red bar at column 72 – any data
beyond here gets truncated in line with mainframe commenting standards.
*
COM function addSimpleFilter() now exports the isUnicode option.
*
Much more complex pattern matches now allowed - increased limit of capturing
patterns from 99 to 65535.
*
Removed environment variable capability from the split filename because of
conflicts with %f, %e and %d macros for the file name, extension and number.

What's New in TextPipe v8.4.9 – February 18, 2010
=================================================

*
Mainframe copybook filter – alpha fields now have quotes added around them,
and existing quotes are doubled up to escape them.
*
Environment variables now allowed in the split filename.

What's New in TextPipe v8.4.8 – December 3, 2009
================================================

*
Added [alphanumeric] and [numeric] to EasyPatterns.
*
Fixed validation when a Restrict to Filename NOT matching filter was missing
subfilters.
*
Fixed bug in invocation from shell extension.

What's New in TextPipe v8.4.7 - November 27, 2009
=================================================

*
Modified TextPipe and TextPipe Engine so that error messages are now
available through the ErrorText variable when called via COM.
*
Output file size was missing from the log.

What's New in TextPipe v8.4.6 - November 24, 2009
=================================================

*
Fixed Unicode Search/Replace error that caused file corruption beyond
64Kbytes.

What's New in TextPipe v8.4.5 - November 20, 2009
=================================================

*
Fixed error with clearing of output strings.

What's New in TextPipe v8.4.4 - November 18, 2009
=================================================

*
Script errors that occur at Run Time are now logged as warnings in the log
rather than stopping the filtering.
*
Fixed error with parsing of output strings.

What's New in TextPipe v8.4.3 - November 5, 2009
================================================

*
Fixed shell extension not starting filters correctly.

What's New in TextPipe v8.4.2 - October 29, 2009
================================================

*
New 'Prepare Query' option for database filter to make SELECTs more
compatible.
*
Fixed elapsed time display when duration was less than 1 second.
*
New Edit Menu\Preference to control if Word Wrap is on by default in edit
controls.
*
New command line option /fromshellext to indicate if a filter is being run
from the Windows Explorer shell extension. Existing files in a filter
triggered from Windows Explorer are now cleared and a warning given.
*
Added new COM function addMainframeCopybookFilter() to recreate Mainframe
Copybook Filters.
*
New installer should allow overwrite of existing installs and/or uninstall
without reboot.

What's New in TextPipe v8.4.1 - October 13, 2009
================================================

*
Files to Process, and Search and Replace List filenames can now be specified
relative to the current filter's path.
*
Improvements to Customize Shell Extensions dialog.
*
New option for Grep filter to ignore empty matches.

What's New in TextPipe v8.4 - October 6, 2009
=============================================

*
New pre-compiling of replace expressions for faster replacing.
*
New 'Break on Change' option for captured variables makes debugging far
easier.
*
Revamped the simultaneous search/replace filter to work with lists of
300,000+ replacements. New Longest First option.
*
Copybook filter now copes with nested OCCURS where the nesting contains only
one field (special case).
*
Copybook filter now copes with OCCURS DEPENDING fields.
*
Copybook filter now outputs the header only once - very useful inside the
restrictions that are required for processing variable format files.
*
Copybook wizard - fixed disabling of trailing delimiter on last field.
*
New command line option /CLOSEONCOMPLETION - which closes the filter window
when a thread completes. This is typically used with a /Z command line file
and a large set of filters.
*
New option for Count Duplicates Filter to change the format of the results -
e.g. put tab between the count and the string, reverse the order etc.
*
Grep filter now has an option to match UTF-8 characters for Unicode text.
*
Convert to Word List filter now allows words with numbers such as Catch-22,
24-7, 5th, 7-Eleven etc.
*
Remove duplicate lines AND Count duplicate lines filters now do not require
that the duplicate lines be adjacent (i.e. that the list is pre-sorted).
However, it now has the restriction that it cannot speedily remove or show
duplicates from files LARGER than main memory.
*
Grep filter has now been changed so that it does NOT match empty lines.
*
TextPipe is now much faster when logging is disabled.
*
TextPipe no longer records output file names beyond a count of 10000. These
filenames are only used for .BAK file recovery, and they cause a performance
slow down when millions of files are being processed.
*
Fixed function for removing empty output files. The input file was being
deleted if the output file had zero size.
*
Fixed problem with duplicate data appearing in large sorts requiring merge
files with 2 or more merge stages.
*
Reverted handling of blank fields/blank lines to pre 8.3.5 handling. Pad to
Width and Add Left Margin filters cannot be used for blank fields (use an Add
Footer filter to give a blank field some width first).
*
Fixed bug in HTML Restriction filter where the action of the 'Include start
and end tag' option was inverted.

What's New in TextPipe v8.3.7 - July 17, 2009
=============================================

*
New Simultaneous search replace option for Filters\Search\Search and Replace
List Filter- use when your search and replace terms overlap each other e.g.
replace 1->7, 7->10, 10->1.
*
Copybook filter now recognises field names starting with numerics e.g. 14B-
DATE.
*
Copybook filter now copes with nested OCCURS, and numbers the generated field
names correctly.
*
New 'Copy to Clipboard' option when a hex dump is displayed from a field's
context menu.
*
New Output filter option to remove empty output files (those with a size of 0
KB).

What's New in TextPipe v8.3.6 - June 24, 2009
=============================================

*
Fixed performance of Trim option of mainframe copybook filter.

What's New in TextPipe v8.3.5 - May 5, 2009
===========================================

*
Fixed handling of blank fields when using Restriction Filter and a subfilter
such as Pad to Width, or Add Left Margin.
*
Fixed file modification detection when using subfilters of a Search/replace
or HTML Restriction filter.
*
Fixed memory leak that occurred when the Skip Binary files option was used.
*
Fixed VBScript/JScript code generated - missing end of line for
TPWindow.inputDeleteFiles.

What's New in TextPipe v8.3.4 - April 23, 2009
==============================================

*
Fixed bug with [Delete BAK files] button not being enabled.
*
Installer changes to support Vista better.

What's New in TextPipe v8.3.3 - March 27, 2009
==============================================

*
Fixed bug with Converting Excel spreadsheets to text when the output was set
to the clipboard.
*
If logging is enabled, a setting of 'Don't log to file if line count is less
than' = 0 now causes all lines to be logged, rather than none. To disable
logging, uncheck the logging flag. This primarily affects calling filters
from the TextPipe Engine.
*
The copying of all predefined global variables from the command line
(@name=value) now occurs no matter how the filter is called.
*
Enhanced Windows Explorer Context Menu handler to better handle huge sets of
files (20,000+).

What's New in TextPipe v8.3.2 - March 5, 2009
=============================================

*
Fixed bug in using UNC paths \\server\share etc - these conflicted with the
environment variable parser's desire to eliminate duplicate \\
*
Changed Directory scanning routines to allow files with any attributes.

What's New in TextPipe v8.3.1 - March 2, 2009
=============================================

*
Fixed bug in using external reference file for Replace->Search/replace list

What's New in TextPipe v8.3 - February 24, 2009
===============================================

*
Now recognizes the VALUE clause in mainframe copybooks.
*
EasyPatterns CSVfield, TABfield and Pipefield now handle unquoted fields
correctly.
*
Macros can now include environment variable values e.g. @ENV_OS inserts the
text 'Windows_NT' on an NT system.
*
Filenames specified in TextPipe can now make use of embedded environment
variables using e.g. %PROGRAMFILES%, %TEMP% syntax.
-	Search/replace list filter
-	Log filename
-	Execute filter input/output filename and command to run
-	Sound to play on completion, and default editor
-	Add Left/Right Margin, Header/Footer, Add Text Side by Side
-	Output filter new extension, new output folder and single output
filename
-	Merge and Link filters.
*
New Output Filter option for No Output - the filtering is used for its side
effect only.

What's New in TextPipe v8.2.3 - December 16, 2008
=================================================

*
Now recognizes the new Office 2007 format .xlsx (e.g. search/replace lists).
*
Fixed line count statistics displaying -1 for subfilters that are never
invoked.
*
Default is now to process compressed files (.zip, .docx, .xlsx, .pptx).
*
Restored play sound on completion for command line filters.

What's New in TextPipe v8.2.2 - December 15, 2008
=================================================

*
Removed playing a sound in non-interactive mode (from command line or from
COM).
*
Interactive error messages are now disabled when TextPipe is called from the
command line or from COM (when TextPipe.visible = false).

What's New in TextPipe v8.2.1 - December 9, 2008
================================================

*
Removed file copy on first run from Engine version.

What's New in TextPipe v8.2 - November 25, 2008
===============================================

*
New option to allow/prevent processing of files inside Compressed files such
as .ZIP, .DOCX, .XLSX and .PPTX.
*
[Copy lines to log file] button now also adds a log message to make the log
file easy to open.
*
Right-clicking the log results now shows an option to copy the log lines to
the clipboard.
*
Clearer warning for interactive logging ending.
*
Fixed EasyPatterns for CSVField, TabField and PipeField to avoid stack
overflows.
*
'Include the start and end tag' option of HTML filter was reversed.

What's New in TextPipe v8.1.12 - October 21, 2008
=================================================

*
'Output Filter\Open output on completion' now opens all output files, not
just the last one.
*
Better handling of copybook Comp3CanBeUnsigned option.
*
Now traps numeric conversion errors when sorting floating point values such
as 1e-9999.
*
Word concordance now has correct default for the output format, and the found
word is now correctly positioned in the context, and capitalized.

What's New in TextPipe v8.1.11 - October 6, 2008
================================================

*
Fixed buffer underrun problem with search/replace filter, where maximum match
sizes larger than 4KB were not correctly performed.

What's New in TextPipe v8.1.10 - September 8, 2008
==================================================

*
New 'Paste File List from Clipboard' button on Files To Process tab.
*
Files to Process grid is now refreshed after loading a File List.
*
Embedded Help is now loaded correctly from the application folder.
*
URLs can now be inserted or loaded from a File List without being interpreted
incorrectly as filenames.

What's New in TextPipe v8.1.9 - August 26, 2008
===============================================

*
Fixed flushing of T-filter child filters.

What's New in TextPipe v8.1.8 - August 25, 2008
===============================================

*
Added new 'T-filter' to copy the data and process it in a different branch
without affecting the main branch.
*
Changed Unicode filter Unicode to Escaped ASCII to also convert Latin (128-
255) characters to escaped \Uxxxx format.
*
Improved right-clicking in filter tree to first select the node that is
right-clicked on.
*
Improved handling of invalid zip files.

What's New in TextPipe v8.1.7 - July 19, 2008
=============================================

*
Fixed problem loading replace filters from versions 7.8.3 and older with \@
or \$ in the replacement text.

What's New in TextPipe v8.1.6 - July 9, 2008
============================================

*
Added new filter to Restrict to UTF-8 files.

What's New in TextPipe v8.1.5 - July 2, 2008
============================================

*
Fixed bug with named parameters in search/replaces - if a subexpression was
not captured then any remaining named parameters would have no value and
corresponding global variables would not be set.

What's New in TextPipe v8.1.4 - June 24, 2008
=============================================

*
Clipboard output now shows a non-zero size.
*
Statistics are now reset correctly when a filter has subfiters.
*
Copy to log button now correctly re-enabled when a job starts for the second
time.

What's New in TextPipe v8.1.3 - June 16, 2008
=============================================

*
Improved internal patterns used by the HTML filter to prevent stack overflow
errors.

What's New in TextPipe v8.1.2 - May 27, 2008
============================================

*
Fixed bug where /Q and /MINIMIZED were not recognised when placed in a /Z=
command file.
*
Fixed bug where subfilters of Restrict to Matching lines were not called when
a count of matching lines was being output.
*
Fixed discrepancies between docs, TextPipe and TextPipe Engine for
addSecondaryOutputFilter() function, output options and for filenames in the
File Grid.

What's New in TextPipe v8.1.1 - April 29, 2008
==============================================

*
Fixed error using a Search/Replace action of 'Send variable X to subfilter'
when the previous replacement text had been set to '$0'.

What's New in TextPipe v8.1 - April 21, 2008
============================================

*
Improved handling of non-DOS EOL files - TextPipe now checks if your filter
has an EOL Conversion filter before offering to add one for you.
*
Link filters can now use a path relative to the current filters path
(previously this was relative to the TextPipe installation path). This makes
them far easier to link to.
*
The Capture Filter now allows variable names containing digits.
*
Globals can now be set on the command line using the syntax @name=value with
a space between each set of values.

What's New in TextPipe v8.0.9 - March 25, 2008
==============================================

*
New extended logging for Remove Blank Lines, Remove Blanks from Start/End of
Line.

What's New in TextPipe v8.0.8 - March 18, 2008
==============================================

*
New notice level message if MS Word or Excel files are processed using the
evaluation version.
*
If Remove/Restrict Delimited Field filters remove fields from the end of the
line, TextPipe now removes any hanging delimiters.
*
The Convert Excel files to Text filter now converts all sheets, not just the
first one.

What's New in TextPipe v8.0.7 - March 5, 2008
=============================================

*
Zoned decimal conversions now allow positive and negative signs.

What's New in TextPipe v8.0.6 - February 20, 2008
=================================================

*
When run from the command line, TextPipe now does not show a dialog on errors
such as a filter file could not be loaded. An exit code is set and the log
file written instead.
*
Fixed a bug where the Analyse function on the Files Grid did not return a
result.

What's New in TextPipe v8.0.5 - February 14, 2008
=================================================

*
Fixed a bug where the Copybook Filter used the wrong parser if PIC was
preceded by a tab instead of a space.
*
Copybook Filter now copes with 'PIC GROUP' syntax.

What's New in TextPipe v8.0.4 - February 13, 2008
=================================================

*
Capture filter can now resets at the beginning of the file, at the beginning
of each region or both.

What's New in TextPipe v8.0.3 - February 4, 2008
================================================

*
Right-clicking a filter now shows a 'Help on this filter' link which quickly
takes you to the help file without needing to add the filter. Using
Help\Embedded Help also constantly updates to show you help on the current
filter.
*
The Status tab is now displayed by default when filters are run from the
command-line or from Windows Explorer.
*
Mainframe filters now insert implied decimals correctly when the fields are
blank padded.
*
Folder overload prevention for the split filter now triggers at 1000 files
(was 10000).

What's New in TextPipe v8.0.2 - January 9, 2008
===============================================

*
Now warns if the user tries to use the Trial Run area to process a Mainframe
Copybook (a file must be used).
*
Task Menu filters now open a status window so that progress and/or errors can
be reviewed.
*
The evaluation version (even an expired version) can now be used to check if
a filter is compatible with the Pro, Standard or Lite version.
*
The Restrict to Filename filter (used to rename files on the fly) now skips
processing filenames inside zip/docx/xlsx/pptx files.

What's New in TextPipe v8.0.1 - October 30, 2007
================================================

*
Fixed implied decimal inserted in wrong position for zoned decimal fields.

What's New in TextPipe v8 - October 24, 2007
============================================

*
The Status Window now forms a new tab of the Filter Window - making it easier
to refer to results. The results log is now also far easier to read.
*
The Mainframe Copybook Filter (new) and Wizard (deprecated)
- now handle OCCURS without TIMES when the repeating field is an unnamed PIC.
- now handle masks like PIC 9999/99/99 - these are treated just as an Alpha
field, in this case, PIC X(8). No mask substation is performed.
*
The Mainframe Copybook Filter now generates numbered field names when the
fields are inside a repeated group. This wizard does not do this, but the
field header can be manually edited.
*
Improved the EasyPatterns CSVfield, TABfield, PipeField, HTMLtag,
HTMLstartTag, HTMLendTag, QuotedString so that they are much faster.

What's New in TextPipe v7.9.11 - October 24, 2007
=================================================

*
Fixed bug with Search/Replace in Exact mode, if the Action was set to
'Remove', the text was not actually removed.
*
Added collection and logging of statistics to Lite and Standard editions.

What's New in TextPipe v7.9.10 - October 19, 2007
=================================================

*
Log filenames are now displayed with an absolute path to make them easier to
locate. When [Copy Lines to Log File] is clicked in the Status Window, now
ALL lines are copied.
*
Fixed handling of invalid .zip files.
*
Convert MS Word/Excel documents to text - better handling of MS Word/Excel
failures.
*
Improved status dialog with single tab display.
*
Log files now combine date and time into single field.

What's New in TextPipe v7.9.9 - October 8, 2007
===============================================

*
Fixed output of implied decimal values for Mainframe Copybook conversions.

What's New in TextPipe v7.9.8 - September 27, 2007
==================================================

*
Fixed binary file detection logic.

What's New in TextPipe v7.9.7c - September 18, 2007
===================================================

*
Fixed handling of last file in compressed Office 2007 formats (docx, xlsx,
pptx).

What's New in TextPipe v7.9.7b - September 12, 2007
===================================================

*
Fixed script generation for addUnicodeConversionFilter2.
*
Fixed COM function TextPipeFilter.compiledFilterStartJob.
*
Grep functions can now match blank lines using the pattern ^$.
*
Fixed Copybook Filter choosing incorrect parser if 'pic' used rather than
'PIC'. Copybook Filter now accepts field names that include spaces. Parse
tree output now includes periods so that it is still a valid copybook. Fixed
field width for P (Packed) type when using alternate style copybooks.

What's New in TextPipe v7.9.7 - September 4, 2007
=================================================

*
The File Input's Binary detection now correctly identifies Unicode files, Zip
files and Microsoft Office 2007 file formats (.DOCX, .XLSX, .PPTX) as non-
binary.

What's New in TextPipe v7.9.6a - August 29, 2007
================================================

*
Debug Mode of Copybook Filter now shows field length and start and stop
columns. Copybook Filter now handles OCCURS after the PIC definition.
*
New internal editor for reviewing text files.

What's New in TextPipe v7.9.6 - August 24, 2007
===============================================

*
Zip files - now processes and updates data stored inside compressed files
such as .zip, and Microsoft Office 2007 file formats such .DOCX, .XLSX, .PPTX
etc.
*
Word conversion filter now clears previous results in case of error, and
disables macros during run.
*
Removed 64k limitation on string data saved to registry, fixed error with
string data over 64k not being allowed.

What's New in TextPipe v7.9.5 - August 10, 2007
===============================================

*
Fixed Width Wizard now accepts both
  <field name>   <column width>     and
  <field name>   <start column>-<end column>
formats.
*
Now uses updated pattern match library to resolve regex problems.

What's New in TextPipe v7.9.4 - August 1, 2007
==============================================

*
New status window option to Delete backup files (in addition to restoring
backup files). Status window now warns you if you attempt to close it before
dealing with backup files.
*
Better warning message for invalid file patterns in the Files to Process
grid.
*
Copybook REDEFINE clause now allows an optional trailing period.

What's New in TextPipe v7.9.3 - June 22, 2007
=============================================

*
New Word Concordance filter under Filters\Convert.
*
New filter Restrict to Blocks of Lines - useful for adding text after every
XX lines.
*
Reinstated Filters menu for accessibility, and File Menu items Link to
Filter, Insert a Filter, Search for a filter.
*
Refined the pattern generated by filter Restrict to Between Tags.
*
Script filter now works with PerlScript engine - all function validation is
disabled.

What's New in TextPipe v7.9.2 - June 12, 2007
=============================================

*
Added semi-colon (;) as an output format for the Mainframe Copybook Wizard
and Fixed Width Wizard.
*
Added error for blank text in the Mainframe Copybook Filter.
*
Fixed incorrect length detection for signed zoned decimal in the Mainframe
Copybook Wizard (the Copybook Filter used the correct result).
*
Reinstated wizard on startup.

What's New in TextPipe v7.9.1 - May 31, 2007
============================================

*
Fixed pattern generated by filter Restrict to Between Tags.
*
The default filter Normal.fll is now loaded from the Documents folder
regardless of which folder TextPipe is started from.
*
C:\ is now not used as a temporary folder, instead it is requested from
Windows.
*
Maps are now retrieved from the Documents folder.
*
Code page functions are now unavailable under Windows NT.
*
New prompt for language on first run.

What's New in TextPipe v7.9 - May 28, 2007
==========================================

*
New Multi-Processor support (in Server version) with Filters\Special\Switch
CPUs to direct filtering to different CPUs. This enables massive performance
enhancements for large data crunching tasks.
*
Vista install support.
*
Now detects Unix files being loaded into Trial Run area, and warns the user
and offers to add an End-Of-Line conversion filter.
*
Mainframe Copybook Wizard - fixed incorrect implied decimal.
- has been completely re-written as a filter. The parser now shows the line
number of any parse error, handles INDEXED BY clauses, logs warning if
invalid record at end of file, XML output option.
- now handles non-PIC style copybooks
*
TextPipe now highlights the filter in error correctly, taking into account
comment filters.
*
New macro @filterFilename to output the filter filename.
TextPipe.filterFilename is also available in the scripting filter.
*
Comment filters are now shown in green italics to make the filter list more
readable.
*
Fixed pattern generated by filter Restrict to Between Tags.
*
An inability to log messages to the log file now causes a Windows Event Log.
*
Tightened up the Analyse File Format function to correctly identify BOM and
DOS End-Of-Lines in files.
*
Fixed intermittent /Q Exit On Completion switch. This only occurred on fast
CPUs when a new processing thread started and finished before (!) the main
thread processed the /Q command.
*
Fixed EasyPattern bracket optimization. Now shows the line number of any
error, and fixed [PhoneNumber] pattern.
*
Fixed Perl/EasyPattern exception with named captured variables and Prompt on
Replace was turned on, when the named captured variable did not match
anything.
*
Removed remaining memory Info log item on Merge Sort.

What's New in TextPipe v7.8.13
==============================

*
Mainframe Zoned Decimal expansion now allows negative to be coded with '2' or
'5'.
*
Comment and Scripting filters now get display of special characters turned
off by default.
*
Mainframe Wizard now removes the trailing delimiter on the last field.

What's New in TextPipe v7.8.12
==============================

*
Explorer Context Menu now shows the filename without the extension.
*
Convert Excel to CSV filter now does not use a TEMP file in C:\
*
Mainframe Copybook Wizard now
- correctly discards INDEXED BY statements.
- removes the FILLER header associated with removed FILLER fields.
- VALUE lines are now ignored rather than causing a warning.
*
Windows from TextPipe.Alert calls now allow the filter to be cancelled.

What's New in TextPipe v7.8.11
==============================

*
Fixed bug with handling of Replace Action of 'Send $x to subfilter'.
*
Fixed bug with HTML filter removing the Restricted text.

What's New in TextPipe v7.8.10
==============================

*
New 'Action' field for Search/Replaces, with the default being just
'Replace', as before. The new field allows the matched value to be Removed,
to be sent to subfilters, or send just part of the matched value, such as $1-
$9. It also allows NON-matching text to be subfiltered, making it very easy
to process text outside bracketed expressions like <?php ... ?>
*
New Output Filter options:
- Only Output Modified Files has been extended to allow the original files to
be deleted if they have been changed. This is very useful for repeated
processing of files to different output folders.
- Ignore Initial Folder - when the folder structure is being maintained, this
allows the start folder to be removed from the destination folder name.
*
EasyPatterns now support named capturing using the syntax:
  [ capture( ... ) as 'VarName' ]
*
Optimized the Head/Tail filter when Including Bytes from the End of a file.
*
The Filters Remove Lines From List and Extract Lines From List now do not
recognize special characters such as \. This is so that lists of filenames
with backslashes can be pasted in.
*
File Grid 'Ignore' option renamed to 'Disable'. File Grid now allows files to
be excluded by perl regex pattern or EasyPattern.
*
Replace List filter now allows Find/Replace strings to be Swapped, and the
list to be reversed - very useful if you apply changes to your data and then
reverse them at the end of the filter list.
*
Fixed bug with the Restrict to Filenames Matching Pattern filter
-	the filename was not being passed to subfilters, so double-level
filename restrictions did not work
-	the patterns were incorrectly unescaped before being parsed, so
patterns with backslashes were not correctly interpreted.
*
If too much data is sent to the Clipboard, TextPipe now traps the error and
limits the text to 10MB.
*
Fixed bug with Restrict to Lines in Pattern list - the patterns were
incorrectly unescaped before being parsed, so patterns with backslashes were
not correctly interpreted.
*
Added scripting functions TextPipe.fullOutputFilename, TextPipe.FilterWindow,
TextPipe.Application.
*
Added COM functions FilterWindow.Application, FilterWindow.outputFileCount,
FilterWindow.outputFile() and FilterWindow.outputFileTask().
*
Now warns against CSV or Tab data containing ASCII 26 when converting to XML
(remove or change using a map or Search/Replace filter).
*
Fixed EasyPattern for [EmailAddress].
*
Updated Extract\Email addresses filter.
*
Fixed bug in Match Between HTML tags - Match Case was being set to On rather
than using the displayed setting.

What's New in TextPipe v7.8.9
=============================

*
Increased maximum stack size to eliminate Stack Overflow errors.
*
Fixed bug in Capture Text filter where the new captured value was only set at
the end of a file.

What's New in TextPipe v7.8.8
=============================

*
Processing is now suspended when you hit [Cancel].
*
Added TextPipe.exe path to Run... path.
*
Fixed bug - the Mainframe Copybook Wizard no longer tries to replace
delimiters in the input file. It was replacing delimiters even inside packed
decimal fields.

What's New in TextPipe v7.8.7
=============================

*
Fixed parsing of replace expressions including \x24 (a $ sign).
*
Mainframe Copybook Wizard new options - Remove Filler fields, Include header.
The Wizard now records option settings in the Comment output.

What's New in TextPipe v7.8.6
=============================

*
Fixed sort function not working when start column <> 1 and length < 4096.
*
Extract email addresses now ignores '&'.

What's New in TextPipe v7.8.5
=============================

*
Loading a search/replace list from Microsoft Excel now allows Unicode values.
*
The grep filter now always reports the output file as being 'modified',
preventing confusing behaviour with the 'Only output modified files' Output
Filter option. A change of filename from input to output now also results in
a file being written, regardless of any changes.

What's New in TextPipe v7.8.4
=============================

*
Mainframe Copybook Wizard now handles PIC fields both before and after OCCURS
xx TIMES for single field OCCURS statements.
*
Optimized tree updating as field data is changed.
*
Changed install paths for Windows Vista.

What's New in TextPipe v7.8.3
=============================

*
New internal function TextPipe.finishedWithFile() to halt any further reading
of the current input file.
*
New TextPipe.Application function isBinaryFile() to detect if a file is
binary or not.
*
Updated memo editor and pattern matching engine to fix bugs in third-party
components.
*
Fixed EasyPattern handling of EBCDIC() function.
*
Fixed hex dump filter outputting the last line twice.
*
Fixed handling of Compress to EBCDIC Packed Decimal.
*
Fixed bug converting old replace patterns of form $1$2$3 to new form of
$1$$2$$3.

What's New in TextPipe v7.8.2
=============================

*
New filter to Add a Line at a given line number.
*
Fixed bug with EasyPatterns not returning all of a captured expression.
*
Unicode conversion filter now makes use of the 'error' character.
*
Mainframe copybook filter now copes with INDEXED BY statements, and better
handles nested OCCURS.

What's New in TextPipe v7.8.1
=============================

*
Escaped Unicode now outputs \uXXXX instead of \u+XXXX.

What's New in TextPipe v7.8
===========================

*
New option to show special characters such as spaces, tabs and line feeds.
*
New option - triple-clicking a line selects the whole line.
*
New filter to convert Filters\Convert\UTF16-LE to Escaped Unicode (ASCII +
UCN \U) e.g. \u402d\u403d
*
The file list can now contain http://, ftp:// and gopher:// urls to download
directly.
*
The user is now given the option to manually start Word or Excel if they are
not running when TextPipe needs them (with the Convert Word/Excel Document to
Text filters).
*
Pattern matches can now name captured variables for use in the replacement
text e.g. searching for (?P<dollars>\d+) would then allow the macro @dollars
to be used in the replacement, and also as a global anywhere else in the
filter list.
*
Far more complex patterns can be processed, with support for up to 99
captured variables, from $1-$99.
*
New 'File\Send to...' function for filter lists.
*
New Fixed Width to Delimited Output wizard - makes converting fixed width
files to CSV, Tab simple.
*
Mainframe Copybook Wizard now allows underscores (_) and single quotes (') in
field names in  'XXX_XXXX 4 AN 1-4'-type copybooks.
Wizard now removes the content of fields called FILLER*.
Wizard now copes with repeated single-fields, as well as GROUPED OCCURS.
EBCDIC packed decimal fields can now have either a leading sign, trailing
sign or no sign.
New option to trim leading spaces and zeroes from fields.
New option to only expand numeric fields - Alpha fields are left unchanged.
This is useful when the target application can accept EBCDIC data but can't
handle packed decimals.
New option to allow COMP-3 to be unsigned.
Wizard can now cope with removing line numbers and commented copybook lines
(those with a '*' in column 7).
Wizard can now cope with field definitions spread over multiple lines.
Wizard now remembers settings between uses in a single session.
Expand Binary Number to EBCDIC filter now outputs a fixed width result.
Expand Unsigned EBCDIC filters now output a space for invalid digits A-F.
*
EasyPatterns now produce optimized perl regex output. Added EasyPattern
EBCDIC('') for matching EBCDIC values. Fixed parsing error with ASCII()
function. Now handles back references to groups 1-99. Fixed DayNumber
insisting on leading zero. EBCDIC() and ASCII() functions can now be used in
conjunction with not ... as character classes.
*
Hex displays now all use '.' to replace space.
Query replace window now shows hex offsets for Hex Dump display.
*
Convert Word Document to Text/Convert Excel Spreadsheets to Text now support
Unicode output as an option. Follow this with Convert UTF-16 to UTF-8 to
process these files using normal filters.
*
New File Shred filter to completely overwrite file contents. Run repeatedly
for more shredding.
*
Unicode property tables for pattern matching updated to Unicode 4.1.0, and
matching engine updated to PCRE 6.4.
*
Split filter now can split at a pattern.
New Split Filter option to avoid overloading directories. When more than
10000 files are created, TextPipe starts a new folder.
Fixed bug where \\ could appear in split output filename.
*
Filter list tree now auto-scrolls.
Filter list tree now allows mass enable/disable of filters with select and
right-click.
*
New Filters to Add File Headers/Footers. Unlike the existing Header/Footer
filters, these filters only output once per file even when included inside a
restriction.
*
Now can drag and drop filters between filter lists in different windows.
*
Filters Add Header/Footer/Left Margin/Right Margin now always check for
@macros embedded in the text, regardless of whether the text comes from the
memo or an external file.
*
Fixed bug with search/replaces where an internal conversion from ansi to
Unicode and back could result in high-ansi characters not being mapped
correctly back to their original characters. In particular, the ansi (tm)
symbol mapped back to TWO characters.

What's New in TextPipe v7.7.4
=============================

*
@macros are now allowed in the replace field of a search/replace.
Note: If you load a filter from an older version, '@' will be escaped to
'@@'. The Confirm Replace dialog now shows the values of all global and user-
defined @macros.
*
Warning for Excel or Word documents now disabled.
*
Fixed Seller's Edit Distance search/replace not working.
*
Fixed bug in Mainframe Copybook Wizard for length of signed Packed Decimal
fields.

What's New in TextPipe v7.7.3
=============================

*
No longer deletes current output file when processing is cancelled and the
file is being appended to.

What's New in TextPipe v7.7.2
=============================

*
TextPipe now works with DataPipe successfully.

What's New in TextPipe v7.7.1
=============================

*
To prevent accidental overwriting of files, the default path for the Output
Filter's Single File Output, and the Merge Filter path are now set to blank.
*
Removed the support for quoted tab data. Tab data does not need quotes!
*
Fixed error with deleting filters.
*
Fixed error entering custom delimiter when '\t' was entered.

What's New in TextPipe v7.7
===========================

*
Ultra-fast sorting of huge files (100MB in less than 30 seconds); control
over maximum memory used - for shared servers.
*
Convert Excel spreadsheets to CSV format.
*
Control which filters are applied to which file types with the new Filename
Restriction filter - apply filters to say *.doc files, or *.xls etc.
*
Handle more types of Mainframe data than ever before, with the improved
Mainframe Copybook Wizard:
- discarding line numbers from pasted copybooks
- now handles null packed and zoned numeric fields
- now outputs invalid digits in packed or zoned numerics as their hex
equivalents
- now adds filters to trim leading and trailing space from fields, and
leading '+' signs from numerics
- can now choose output format between Fixed, CSV, Tab- and Pipe-delimited
- now can add leading zeroes to numeric fields where the implied position is
greater than the field width
- output format now includes a record header, which makes it far easier to
validate the output file and import the file into a database
- new filters for unpacking odd/even-length unsigned packed numerics, and for
decoding overlapped unsigned packed numerics
- delimiters are removed from the input file first to prevent extra fields
being created.
*
Easily Extract Text Between Delimiters with the new Web-based Filter Wizard.
*
Review results easily
 - New Output Filter option to open the output file on completion.
 - Open output files easily with clickable links in the Status Dialog.
*
Set the Error Character for Unicode and Code Page conversions when there is
no appropriate character in the target encoding.
*
For precision automation, control TextPipe's pause before exit with the new
/Q=millisecond option.
*
Accept pasted Unix text in the Search/replace filter.
*
Easily match day names, day numbers and arbitrary characters using the new
EasyPatterns [dayname], [daynumber] and [ascii()] function which inserts
arbitrary ASCII codes in hex (eg $ff) or decimal (e.g. 255) format.
EasyPatterns now accept counts and quantities in either hex or decimal
format.
*
Numbers lines across files, with the 'Don't Reset At End Of File' option of
the Line Numbering filter.
*
Count occurrences of all lines, with the 'Include Counts Of 1' option of the
Count Duplicate Lines filter.
*
Debug the output files of External programs using the 'Remove Output Files on
Completion' option of the Execute Program Filter.
*
Store arbitrary text in the Scratch Pad - new Edit\Preference option.
*
Date and time macros now return consistent results across different
localities -
  @time - Inserts the time in the format HHMMSS
  @date - Inserts the date in the format YYYYMMDD
  @datetime - Inserts the date and time in the format YYYYMMDDHHMMSS
*
When the right-click menu 'Save' option is selected, TextPipe now saves the
text exactly as is without interpreting special characters.
*
TextPipe now prompts if you try and exit while a thread is running, and asks
for confirmation if you click [Cancel].
*
When a list of perl patterns is loaded or used, TextPipe now shows the line
number of any error when a pattern parsing error occurs.
*
The grep form's context menu now displays EasyPatterns or perl patterns
according to the pattern type selection.
*
When new filters are added or duplicated they become selected. New filters
are now added with blank fields, and for the Search/Replace filter, the extra
options are set to defaults. After deleting a filter, the next filter is now
selected.
*
The Options panel of the filter tree now scrolls to display buttons that are
not visible.
*
Better handling of files that are missing or open in another process.
*
Fixed Range Check error with optimization of 1:1 maps.
*
Fixed Single File Output - didn't force creation of folders for output file.
*
Fixed Windows Explorer shell extension - was starting the filter one row down
from the correct one.
*
Fixed bug where Extract URLs filter would miss a URL on the very last line.
*
Fixed merge sorting bug (only for very large files) where processing never
ended, and duplicates were not removed.

What's New in TextPipe v7.6.3
=============================

*
New macro @fullOutputFilename. May be blank if the clipboard is being
processed.
*
Fixed deadlock between threads that was slowing down the processing when the
TextPipe.Application object was called from code.
*
Fixed Windows Explorer Content Menu not appearing.

What's New in TextPipe v7.6.2
=============================

*
Fixed bug with grep filters where Match Case option was not recognised.
*
Standardized help text and labels for grep filters.
*
Fixed bug with conversions between ANSI and ASCII.
*
Fixed bug with delimited field parser when it encountered a ' or " in the
middle of the field.
*
Fixed bug with clipboard button not being acted on correctly.

What's New in TextPipe v7.6.1
=============================

*
New filters to Restrict to ANSI files, and Restrict to Unicode UTF16 files
and Unicode UTF32 files. These are exceptionally handy for only converting
ANSI files to Unicode, or only converting Unicode files to ANSI.
*
New filter to convert PDF documents to text format.
*
Menu re-organization: Email menu replaced by new Extract menu, encodings
moved to Convert menu. 'Retain...' items renamed to 'Extract...'.
*
New macros @randomdigit and @randomletter - very useful for generating test
data or for generating random sort keys.
*
New Explorer extension option to Analyse files - detect if they are Unicode,
ANSI, etc.
*
Reverse File performance improvement.

What's New in TextPipe v7.6
===========================

*
New Unicode filters for Upper case, Lower case and Title case.
*
Speed improvements for handling very large files.
*
New Expand All/Collapse All menu on All Filters view.
*
New scripting function to build filters to convert a Mainframe Copybook.
Copybook wizard now inserts a warning for each unidentified field, making
incorrect fields easy to identify.
*
New warning message if no files were found to process.
*
Filters\Convert\Numeric HTML entities To Text filter now converts output
characters to UTF-8. The resulting text can then be converted from UTF-8 to
UTF-16 for display.
*
Menu items for common grep and conversion filters in the Wizard menu. Wizard
menu now is always expanded.
*
Memo and text fields now use the fixed width font.
*
Unicode and code page conversions now do not load the entire file into
memory, they convert the file progressively.
*
Updated Simplified Chinese translation.
*
Help text added to the Unicode/Code page conversion filter.
*
Moved Hex and HTTP encode/decode to the Convert menu.
*
Fixed pasting of text into a memo's Find window.

What's New in TextPipe v7.5.1
=============================

*
Drag and drop now prevents nodes being dropped on filters that do not support
subfilters.
*
Unicode replacement now reverses the order of Match Case to be correct.
*
Reverted help system to old style.

What's New in TextPipe v7.5
===========================

*
New Restrict to lines from list/lines NOT from list and
Restrict to lines matching pattern list/lines NOT matching pattern list.
New Match Case option for these filters too.
*
EasyPatterns for Month and MonthName now do not automatically capture.
*
Mainframe Copybook Wizard now copes with OCCURS where the number of
repetitions is fixed.
*
New filter to convert numeric HTML entities to text e.g. &#183; &#x0D. New
maps to convert text to decimal or hex HTML entities.
*
/CLEARFILTERS now removes all filters except the Log, Input and Output
filters. This retains the default settings from normal.fll
*
/EXTENSION can now be set to blank to disable changing the extension.
*
Fixed Access Violation processing empty result sets.
*
Fixed load file list function.

What's New in TextPipe v7.4.3
=============================

*
Now shows all available filters in a separate tree - these can be drag and
dropped or double-clicked to move them into the filter tree.
*
Fix to Mainframe copybook filter to allow it to recognise field names that
included digits followed by ' A'.

What's New in TextPipe v7.4.2
=============================

*
Fixed bug with Unicode exact match and Unicode pattern match where extra
characters were being output.

What's New in TextPipe v7.4.1
=============================

*
Fixed bug with Unicode exact match and Unicode pattern match where search
terms were not found.

What's New in TextPipe v7.4
===========================

*
Mainframe copybook wizard is now more flexible in the data it matches, and
handles COMP-3 field types, and handles numeric fields with purely decimals
(eg V999).
*
New filters for code page conversion.
*
New filters for Unicode Compose and Decompose:
  NFC - Canonical Decomposition, followed by Canonical Composition
  NFD - Canonical Decomposition
  NFKD - Compatibility Decomposition
  NFKC - Compatibility Decomposition followed by Canonical Composition
  And Canonical Compose.
*
Now supports Unicode command line parameters.
*
Now supports single quoted CSV data, quoted tab data, and backslashes in
quoted CSV or Tab data. Existing text delimiters are now preserved on
conversion between CSV and Tab formats. The EasyPattern [tabfield] can now
match quoted tab data.
*
The Restrict to HTML filter can now set the maximum match size - to match
very large sections of HTML or XML.
*
Clarified examples in perl pattern right-click menus.
*
Speed optimization for interactive search/replace when [Replace Job] chosen.
*
When files are dropped, TextPipe defaults files to 'no subfolders'.
*
Simplified the Trial Run button - now it only performs a trial run, and does
not toggle back to the Filters to Apply tab.
*
XSLT filter now correctly flags modified text - so output files get created
when the 'Only Output Modified Files' flag is set. XSLT filter now reports
the error code and error reason to the log if an error occurs.
*
Run External Program filter now reports in the log if the external program
has a non-zero exit code. The output file handle is now released on error.
*
The Unicode Map filter's 'Replace With Value' string now supports special
characters.
*
Encode Hex filter now does not add line feeds.
*
Right-click option Show Hex Dump now checks if the underlying field is
Unicode or not before displaying data. A toggle between UTF16-LE and UTF16-BE
is also available.
*
Reduced memory requirements of Database filter when SELECTing data sets.
*
Fixed database filter's handling of VarBinary (BLOB) data.
*
Database filter now handles subfilters correctly for encoding VarBinary data.
*
Fixed bug when referencing a CSV or Tab search/replace list where the
replacement text was missing. A new error message now provides the line
number of any errors.
*
Fixed bug with inserting captured text as an HTML table row, and added the
ability to insert as an XML record.
*
Fixed bug with HTML help crashing TextPipe on exit.
*
Fixed bug with Input Filter, Prompt On Binary files setting, if 'No To All'
was chosen TextPipe skipped ALL remaining files, not just binary files.
*
Fixed Unicode display of extended information in the Prompt on Replace
dialog. Note: Unicode patterns do not support capturing- only $0 works.
*
Fixed Map filter not detecting Reset and Clear changes.
*
Fixed help file opening twice when F1 pressed.
*
Fixed mismatch of 16 and 32-bit Unicode strings when converting from ANSI or
ASCII/OEM to Unicode and between code pages.

What's New in TextPipe v7.3.1
=============================

*
Restored registry settings required by Offline Explorer.
*
Changed default for re-loading previous filter to Off.
*
Better exception handling for XSLT filter.

What's New in TextPipe v7.3
===========================

*
New Mainframe Copybook wizard makes it incredibly easy to build a filter to
process mainframe data. Handles packed, zoned, binary and display field
types.
*
New filters to Expand Binary Number To ASCII and Expand Binary Number To
EBCDIC.
*
Global dialogs now make use of popup menus to insert special characters etc.

What's New in TextPipe v7.2.1
=============================

*
New filter to capture text to a global variable. This is much faster than
using the Script filter to capture and then output text - which is the most
common use of the Script filter.
*
New grid component for the Map filter - now supports far better range and
mouse selection.
*
New option to automatically reopen the last filter.
*
The Replace Query dialog now has an extra tab to show a hex dump of the found
text. This can be useful for debugging.

What's New in TextPipe v7.2
===========================

*
Right-click menu for Replace field now provides easy insertion of captured
data as Tab or CSV-delimited fields or HTML.
*
The Prompt On Replace dialog now displays the length of each captured
expression.
*
EasyPatterns 1.9.1 - added [PipeField] and [Pipe] keywords.
*
Fixed error when manually choosing the Skip option to not replace text in a
DBCS file - more than one character was added. Only occurs under DBCS
versions of Windows (eg Chinese).
*
New error message when there are too many capturing expressions in a pattern.
*
Fixed bug with 'First row is header row' not being recognised for processing
delimited data with embedded carriage returns.

What's New in TextPipe v7.1.7
=============================

*
Added short cut menu items for converting between Unicode/ANSI and
Unicode/ASCII.
*
Added line number and offset properties to the TextPipe object available in
the scripting filter.
*
Map filters now have a comment property to describe what they are doing.
*
EasyPatterns 1.9 - added keywords for [number, numbers, leftAngle,
rightAngle, lessThan, greaterThan ]
*
Fixed restriction filters not supporting Secondary Output filters properly.
*
Fixed exact match filter not properly supporting Extract option, and not
correctly handling text when 'Skip' was chosen.

What's New in TextPipe v7.1.6
=============================

*
Search/replace lists now detect inserts, deletes and moves as changes. The
order of moved rows is now retained.
*
Fixed restriction filters not supporting Secondary Output filters properly.
*
Fixed EP keywords for [CRLF], [DOSNewLine], [EmailAddress] and [Hyperlink]
*
Fixed width font for Trial Run Input/Output is now correctly loaded.

What's New in TextPipe v7.1.5
=============================

*
EasyPatterns now allow repetition to be specified using the forms [2..3 ...]
and [2-3 ...] in addition to [2 to 3 ...].
*
Right-click menu item Escape Special Characters now works for EasyPattern
fields.
*
Fixed format string error on filter export.
*
Fixed Search/Replace list -caused an error when it was opened.
*
Fixed Remove Byte Order Mark (BOM) filter when handling UTF-8 files.

What's New in TextPipe v7.1.4
=============================

*
Added right-click menu to EasyPattern fields to make designing patterns
easier.
*
Added Line, Column and Length indicators to Search/replace filter.
*
A warning is now shown for trailing whitespace in the search text, which
often results in the text not being found.
*
Added 5 second countdown before TextPipe exits due to /Q command line
parameter.
*
Greatly improved performance for massive numbers of files- TextPipe no longer
checks for duplicate filenames.
*
Fixed EasyPattern for [month,monthnumber,day,hour] so that they match the
long version of the pattern first, then try the shorter variations.
*
Fixed range check errors with Compress packed/zoned decimal filters.

What's New in TextPipe v7.1.3
=============================

*
Fixed EasyPattern for [word] - previously matched non-word characters.
*
Fixed EasyPattern - '.' was not escaped - now fixed.
*
The Prompt On Replace dialog now shows the perl equivalent to an EasyPattern,
for verification purposes.

What's New in TextPipe v7.1.2
=============================

*
Fixed access violation with changes to the Restrict to Bytes filter.

What's New in TextPipe v7.1.1
=============================

*
Improved Restrict to Bytes (Byte x to End-y) to handle unlimited amounts of
data

What's New in TextPipe v7.1
===========================

*
You can now set the field delimiter for the Restrict To Fields, Copy Fields,
Move Fields and Delete Fields filters. This opens the way for semi-colon-
delimited, #-delimited, !-delimited, ^-delimited and other bizarre data
formats.
*
Added two new filters for compressing mainframe EBCDIC data to Packed Decimal
and Zoned Decimal formats.
*
The Split filter can now set the first filename number.
*
Added new Insert Bytes filter to complement the Insert Columns filter. This
new filter disregards line feeds when it adds text, and is very useful for
inserting commas into EBCDIC text as the text is converted.
*
New documentation links in the Filter Menus.
*
Synchronized scrolling is now properly restored when a filter is loaded.
*
EasyPattern fixes to align with the EasyPattern documentation
- [symbol, symbols] - removed the '!'
- [space, spaces] changed to just matched ASCII 32, not tabs, line feeds etc
- [horizontalwhitespace, hspace] - now only match space and tab
- [verticalwhitespace, vspace, worddelimiter, linedelimiter,
paragraphdelimiter] - now match formfeed
- [anybracket, anybrackets] - now match { and }
*
EasyPatterns now will NOT attempt to combine OR'd negated and non-negated
character sets, such as [ not lineChar or 'a' ]
*
HTML filter now matches tags when the tag name is immediately followed by a
tab or newline, not just a space.

What's New in TextPipe v7.0.8
=============================

*
Fixed EasyPattern handling of vertical tab (\v), which affects the keywords
verticaltab, verticalwhitespace, vspace, worddelimiter, wordchar,
columndelimiter, columnchar, linedelimiter, linechar, whitespace, word, line,
and column. This problem still exists in the perl pattern library, and can be
worked around by using \x0b instead of \v.

What's New in TextPipe v7.0.7
=============================

*
Expand Zoned Decimal and Expand Packed Decimal filters are no longer added
inside a Restriction filter.
*
Changed EasyPatterns for EmailAddress, HyperLink, PageNumber and DayOfYear so
that the constituent parts are captured into $variables for later
substitution.
*
Fixed EasyPattern for FormFeed.
*
Fixed handling of EBCDIC negative zoned decimals.

What's New in TextPipe v7.0.6
=============================

*
Fixed bug with parsing of text with \x{...} characters.

What's New in TextPipe v7.0.5
=============================

*
Fixed shell context items - the last item did not work. Also fixed the path
for the Split File filter.
*
Fixed bug with Rolling back .BAK files - if the Only Output Modified Files
option was on, it would delete original files.

What's New in TextPipe v7.0.4
=============================

*
Unicode conversion filter was only outputting the Byte Order Mark for the
first file.

What's New in TextPipe v7.0.3
=============================

*
Added swap button to Unicode conversion filter.
*
Fixed Unicode conversion filter bug.

What's New in TextPipe v7.0.2
=============================

*
Fixed Backup Mode behaviour when the .bak file already existed (now it is
deleted).

What's New in TextPipe v7.0.1
=============================

*
Fixed Hex Dump option for Edit Fields and File Grid.

What's New in TextPipe v7.0
===========================

*
New Unicode (UTF-16LE) search and replace, with options for space compression
and for ignoring non-spacing characters.
*
New Unicode (UTF-16LE) pattern matching.
*
Unicode conversion filter is now much simpler, as you just choose an input
and output format in one step.
*
New XSLT filter for manipulating XML.
*
New file grid right-click menu items:
- Analyze file - checks for various Unicode, XML, CSV and Tab delimited
formats, Unix/PC/Mac end of line format and other file types
- Hex dump - displays a hex dump of the first 200 characters of the file
*
Added \x{...} Unicode character matching for exact matches. This means that
for a Unicode space you can type \x00000020 instead of \x00\x00\x00\x20.
*
Synchronized scrolling option for Trial Input and Output.
*
You can now control if the Trial Input and/or Output are treated as Unicode.
*
Optimized single byte maps for the special case of one output character for
every input character.
*
Better help contexts for the 9 different search/replace types.

What's New in TextPipe v6.9
===========================

*
Removed restriction on pattern recursion depth - this was causing Stack
Overflow errors for legitimate expressions.
*
Fixed file grid ordering - if you moved rows around, TextPipe did not
remember the new row position.
*
Fixed menu items for Remove, Copy and Move tab fields. These did not add the
required filters.
*
Fixed help hyperlinks on Trial Run and Files tabs.
*
Fixed bugs in Unicode Make Little/Big Endian filters.
*
Fixed bug in Convert From Unicode filter, for when it encountered a character
that could not be converted. It now outputs a space.
*
The EasyPattern [year] keyword now checks for a 4 digit year before checking
for a 2 digit year.
*
Convert Word Document To Text filter now only closes the Active Document,
avoiding the MS Word error message 'This method or property is unavailable
because a document window is not active'.
*
Fixed line wrapping problem with the clipboard export text of several
restrict-type filters.
*
Increased maximum stack size to cope with increased size of Unicode filters
(and avoid Stack Overflows).
*
Fixed database filter not quoting char or date/time data fields.

What's New in TextPipe v6.8.3
=============================

*
New filters for handling pipe-delimited files:
  Remove pipe fields
  Restrict to pipe fields
  Copy pipe fields
  Move pipe fields
*
Added EasyPattern keywords for slash, colon, hash, pound, percent, star,
asterisk and ampersand. EasyPatterns now generate much better perl regexps.
*
EasyPatterns can now set the maximum expected match size - this allows them
to match text larger than the default of 4K.
*
Increased processing speed for Trial Runs and Clipboard output.
*
Restrict to fields filters now export the settings for Process Fields
Individually, Exclude Delimiter, Exclude Quotes.
*
Fixed EasyPatterns [month] and [monthname] to attempt to match a long month
name (January, February) first, then a short month name (Jan, Feb).
*
Fixed bug with Context Grep with lines after the match. Matching lines at the
end of the file were being omitted.
*
Fixed code for handling CSV, Tab and Pipe delimited files with header lines.

What's New in TextPipe v6.8.2
=============================

*
Added new EasyPatterns for [ CSVfield ] and [ TABfield ].
*
New Restrict to Byte Range filter - useful for expanding mainframe packed and
zoned decimal fields.
*
New Delete Byte Range filter.
*
XML filter now has better matching of double and single quoted attributes.
These filters now handle embedded double single or double quotes, quotes
escaped with backslash, and embedded new lines.
*
Optimized expand packed decimal and expand zoned decimal for speed.
*
Fixed bug with swap UTF-16 and swap UTF-32 word order.

What's New in TextPipe v6.8.1
=============================

*
Added Expand packed decimal and Expand zoned decimal filters.
*
Fixed handling of Restrict to between tags filter - under some conditions the
trailing > of the start tag would be put in the replace result.
*
Fixed Copy CSV/Tab fields bug. When copied fields were appended, the
delimiter was not inserted if the copy position was the next available field.

What's New in TextPipe v6.8
===========================

*
Added Unicode functions
  - Remove BOM (Byte Order Mark)
  - Swap UTF-16 word order
  - Swap UTF-32 word order
  - Make Little Endian
  - Make Big Endian
*
Added 144 Unicode conversion functions to convert to and from:

ARMSCII-8       ASCII          BIG5              BIG5HKSCS
C99             CES-BIG5       CES-GBK           CNS-11643-1
CNS-11643-15    CNS-11643-2    CNS-11643-3       CNS-11643-4
CNS-11643-5     CNS-11643-6    CNS-11643-7       CNS-11643-INV
CP1046          CP1124         CP1125            CP1129
CP1133          CP1161         CP1162            CP1163
CP1250          CP1251         CP1252            CP1253
CP1254          CP1255         CP1256            CP1257
CP1258          CP437          CP737             CP775
CP850           CP852          CP853             CP855
CP856           CP857          CP858             CP860
CP861           CP862          CP863             CP864
CP865           CP866          CP869             CP874
CP922           CP932          CP949             CP950
DEC-Hanyu       DEC-Kanji      EUC-CN            EUC-jisx0213
EUC-JP          EUC-KR         EUC-TW            GB18030
GB2312          GBK            GEORGIAN-ACADEMY  GEORGIAN-PS
HP-ROMAN8       HZ             ISO-2022-CN       ISO-2022-CN-EXT
ISO-2022-JP     ISO-2022-JP1   ISO-2022-JP2      ISO-2022-JP3
ISO-2022-KR     ISO646-CN      ISO-646-JP        ISO-8859-1
ISO-8859-10     ISO-8859-13    ISO-8859-14       ISO-8859-15
ISO-8859-16     ISO-8859-2     ISO-8859-3        ISO-8859-4
ISO-8859-5      ISO-8859-6     ISO-8859-7        ISO-8859-8
ISO-8859-9      ISO-IR-165     JAVA              JIS_X0201
JIS_X0208       JIS_X0212      JOHAB             Johab-Hangul
KOI8-R          KOI8-RU        KOI8-T            KOI8-U
KSC-5601        MacArabic      MacCentralEurope  MacCroatian
MacCyrillic     MacGreek       MacHebrew         MacIceland
MacRoman        MacRomania     MacThai           MacTurkish
MacUkraine      MULELAO-1      NEXTSTEP          RISCOS1
SHIFT-JISX0213  SJIS           TCVN              TDS565
TIS-620         UCS-2          UCS-2BE           UCS-2-Internal
UCS-2LE         UCS-2-Swapped  UCS-4             UCS-4BE
UCS-4-Internal  UCS-4LE        UCS-4-Swapped     UTF-16
UTF-16BE        UTF-16LE       UTF-32            UTF-32BE
UTF-32LE        UTF-7          UTF-8             VISCII

*
New context menu item to insert control characters like tabs and form feeds.
*
New context menu item to show a hex dump of the selected text.
*
Added logWarning to scripting model.
*
Added Cut/Copy/Paste to file grid context menu.
*
Added Tile function to Window menu.
*
The Convert Word Documents To Text filter now disables Word's screen display
for faster processing.
*
The TextPipe Engine DLL now has a COM interface identical to TextPipe's, but
removes most GUI operations for incredible speed.
*
Added alert() function to the internal TextPipe object of the scripting
filter, so JScript functions can easily display debug dialogs.
*
EasyPatterns bug fixes and improved flexibility. In particular, the [char]
keyword was fixed.
*
Several filters that should have allowed subfilters did not. Filters affected
were: database filter, insert column filter, add left margin filter.
*
Fixed bug - the filename was being output even for non-matching lines, when
context lines were included.
*
Fixed bug with partial trial run menu items including too many items.
*
Fixed bug with the filter tree allowing nodes to be pasted on the Input
Filter, resulting in error-producing subfilters of the Input filter.

What's New in TextPipe v6.7
===========================

*
EasyPatterns - English-like text patterns for mere mortals e.g. to match
phone numbers, use
  [3 digits, punctuation, 3 digits, punctuation, 4 digits]
As easy as that!
*
Many fantastic new HTML and XML filters!
* Restrict a search/replace or other filters to an XML element, attribute or
to text between XML tag pairs
* Remove an XML element, attribute or XML tag pair.
*
New Restrict To Filename filter. This allows you to search and replace inside
filenames, number and rename files and much more!
*
Very powerful new debugging feature - selective trial runs. You can select
which filters are being tested without having to clone a piece of the filter.
You can perform a Trial Run on:
* Entire Filter List (current behaviour, default)
* Selected Filter Only
* Selected Filter and Sub-Filters
* Filter List up to and including Selected Filter
To see these new options, right-click on a filter in the tree view.
*
New Backup Mode - renames original files to .BAK. Easily restore originals
with a single click in the Results window.
*
New Convert Word documents to text files filter.
*
Fixed bug when the Result window Cancel button was clicked twice in a row.
*
Fixed bug with pasting filter tree items that were copied from level 2 or
deeper.
*
Fixed bug with the Windows Explorer Extension that showed the TextPipe menu
multiple times.
*
Fixed bug with multiple consecutive Script filters not passing header text.
*
Fixed bug with Search/Replace Enable UTF-8 option and Extract option being
swapped when saved.
*
Fixed loading of Unicode maps from .TAB and .CSV formats - blank fields were
erroneously skipped.

What's New in TextPipe v6.6.5
=============================

*
Fixed bug with Remove Duplicates option of Sorting filter not being
respected.

What's New in TextPipe v6.6.4
=============================

*
New macros @time, @date and @datetime.
*
Fixed bug with handling of CSV fields with more than one line embedded in a
column.

What's New in TextPipe v6.6.3
=============================

*
Vastly improved sorting performance for files larger than 50 MB.
*
Simplified user interface by removing
1. Comments tab (old comments are now inserted into the filter list as the
first item)
2. Forcing the MDI child window to initially appear maximized, making it look
like a SDI application. The window size can still be changed as before
3. Removed the title that appeared above the properties
4. Removing the scratch pad tab unless a loaded filter has text in the
scratch area.
*
Added UTF-32 Unicode Maps
*
When Prompt On Replace is on, it can now be disabled during the search. Note
- this does not update the tree view immediately, and if the filter is
currently selected it has no effect.
*
The evaluation version provides a warning if the user attempts to modify
Excel or Word documents.
*
Unicode maps can now append multiple map files together into one map.
*
Unicode maps can now log overlapping regions as warnings or errors.
*
Improved Unicode map timings for populating with Default or blank values.
*
Added multi-maps for mapping input Unicode to multiple output Unicode forms
depending on command line parameters.
*
/MINIMIZED now examined before any other parameter, eliminating flicker.
*
Fix error when the file grid was selected in the askSam version.
*
Fix for Run Time version that prevented filter additions after a filter had
been loaded with /F=.

What's New in TextPipe v6.6.2
=============================

*
New File Grid context menu option to copy 100 Kbytes to the Trial Run Area.
*
Upgraded to PCRE 4.3 pattern matching library to fix assertions not working
correctly.
*
Added warning message that not all filter types can be exported to the
command line.
*
Tips and Wizard dialogs now allow filters being double clicked on from
Windows Explorer to be loaded.
*
Tips and Wizard are only shown if a filter hasn't been automatically loaded.
*
Run Time version now disables sound by default and does not show a tray icon.
*
Filter overlay images are now updated correctly when a disabled sub filter is
deleted.

What's New in TextPipe v6.6.1
=============================

*
Added Exit Codes depending on the highest level of error logged.
*
New /EXPORTFILTERTOLOG command line option.
*
Changes to support TextPipe Engine file processing.
*
Trial Input now preserves Unix EOLs when a file is pasted or loaded.
*
Changed menu structure to make it easier for new users. Removed some icons
from the toolbar.
*
Fixed Access Violations with Remove Lines Matching Perl Pattern filter, and
on some random occasions, with Perl pattern matches.
*
Fixed code generation of setPerl() statement.
*
Fixed an extra blank line being inserted at the start of the next file after
a Unicode file was processed.

What's New in TextPipe v6.6
===========================

*
New filter supporting search/replace lists. The lists can either be specified
in a grid, or they can be loaded dynamically from Excel, CSV or Tab-delimited
file formats at run-time.
*
The file list now supports multiple file specifications separated by semi-
colons eg c:\INetPub\*.html;*.txt;*.htm
*
The Split Files filter can now control how the output files are named.
*
Subtract entries in the file list are now shown in red, and Ignore entries
(comments) shown in grey.
*
The Script filter now uses a Syntax highlighting editor to display VBScript,
JScript, VBA, PerlScript and PythonScript code. It also allows separate
default scripts to be saved for each language.
*
Unicode maps can now be loaded from and saved to Excel/CSV/Tab formats.
Unicode filters are now far faster and can deal with undefined characters -
such as removing them, replacing them with a custom string, sending them to a
VBScript sub filter for specialized processing, or allowing them to pass
through unchanged.
*
The filter tree view now supports Cut, Copy and Paste clipboard operations.
There is also an undo action for the current filter. Filters can also be
pasted into any application as a text export.
*
CSV and Tab-delimited files with CR/LFs embedded in fields are now handled
correctly. New restriction filters can be used to remove the embedded CR/LFs.
*
CSV and Tab-delimited restricted filters can now remove the comma or tab
field delimiters and the surrounding quotes to make it easier for sub filters
to work with field data. Each field's data can now be processed individually
rather than as a concatenation of field data.
*
Byte map lines where the output is different to the input are now shown in
yellow. Byte maps can now scroll directly to the character you press on the
keyboard - which makes finding mappings easier.
*
In the audit log, merge file filters now summarize the changes made by
filters that follow them in the filter list. Split file filters also
summarize the changes made.
*
GUI and internal changes to support the new French translation.
*
New command line options /CLEARCOMMENTS, /CLEARFILTERS, /SAVEFILTER,
/LOGENABLE, /LOGMODE, /LOGTHRESHOLD, /PREPAREFILTERFORDISTRIBUTION,
/SAVEFILTER, /PLAYSOUND.
*
When loading a list of commands from a command file with /Z=<command file
name>, comments can now be included in the file using a leading ';'.
*
Speed improvements for the GUI and for filter execution.
*
Audit logging now shows status for more filters including Scripts and Maps.
The audit log text shown for search/replace filters now matches what is shown
on the screen for ease of reference.
*
Various COM scripting changes to support new features, and to allow DataPipe
to set the current directory.
*
Perl patterns now support UTF-8.
*
Convert end-of-lines filter now handles Mainframe CMS V- and F-format files
(downloaded to PC in EBCDIC format).
*
Removed Rich Edit Memo controls - they caused RTF codes in plain text to be
interpreted as fonts, and incorrectly added extra CR/LFs to Add Margins-type
filters.
*
The file list had a bug where it added file entries more than once if they
matched multiple file specifications in the same group.
*
Fixed bug with loading of single byte maps where the map was not actually
loaded, from both the menu and from the map sub-form. Fixed Reverse function.
*
Unicode Maps - fixed problem where changes to the grid were not detected and
saved.
*
Fixed bug with importing Excel and Tab-delimited search/replace lists.
*
Fixed bug with appending filters from the command line with multiple /F=
appearing in reverse order. Fixed filters such as /Hx not being added.

What's New in TextPipe v6.5.1
=============================

*
New filters - CSV to Tab-delimited, CSV to XML, Tab-delimited to CSV, Tab-
delimited to XML.
*
Fixed error message when egrep pattern matcher encountered a \000 (null).
Only perl patterns can handle nulls.

What's New in TextPipe v6.5
===========================

*
Search and replace lists can be loaded easily using the Load List from
Excel/Tab/CSV formats in the Replace menu.
*
The Task Bar Menu and Map Menu now display sub folders of the TaskMenu and
Maps folders, and sort all the filenames and folder names shown. This makes
it very easy to categorize these maps and filters into folders.
*
If a filter has one or more of its sub filters disabled, a red slash is shown
through its icon in the tree view. This makes it easy to find filters where
disabled sub filters are hidden from view. A disabled filter still has a red
cross through its icon in the tree view.
*
Custom patterns can now be defined in the file custom_patterns.txt in the
TextPipe.exe folder. The patterns can be added through the right-click menus
on every field.
*
Sorting can now handle files of unlimited size. Previously sorting was
restricted to using physical and virtual memory.
*
New sorting options for date and time, date, and time.
*
New right-click option for filter tree view to turn a filter into a sub
filter.
*
Matching/non-matching lines and other grep line based filters can now use
Perl patterns, egrep patterns, MS Word patterns and Brief patterns. The
addGrepFilter COM function has changed to support this.
*
Export to VBScript/JScript/web page now also exports the search replace
options such as greedy matching, allow comments, buffer size etc. The edit
distance threshold parameter has been removed from the addReplaceFilter COM
function, and new functions added for setPerl, setEditDistance and
setBufferSize.
*
Menus and tabs re-organisation, selection filters renamed to restriction
filters.
*
A fixed width font is now used for most memo fields.
*
Improved handling of malformed CSV files.
*
Fixed export of Logging:Append to Log. The wrong value was being exported.
*
Fixed generation of random files of specified size. All files were being
truncated at 8k.

What's New in TextPipe v6.4.1
=============================

*
Added "Content Type" to system registry to help web downloads of .fll files.
*
COM object now registers itself in the Running Object Table (ROT), and frees
itself at the correct times.
*
New COM properties Application.visible and Application.activeFilter
*
Fixed bug with moving nodes with sub filters.
*
The Task Bar menu now correctly runs filters for the second and subsequent
runs - previously the status window stopped it from working.

What's New in TextPipe v6.4
===========================

*
New filter to Remove a line range (in addition to the Head/Tail filter).
*
New filter to Remove CSV fields.
*
New filter to Remove Tab fields.
*
New filter to Move CSV fields.
*
New filter to Move Tab fields.
*
New filter to Move Columns.
*
New filter to Copy CSV fields.
*
New filter to Copy Tab fields.
*
New filter to Copy Columns.
*
File/Open command can now open multiple filters at a time.
*
Drastically improved the speed of 'head' filters and Remove All - they now
skip reading the remainder of the file no matter how large it is (a 'Head'
filter is used to extract the first N lines of a file).
*
New option to display help in an embedded window that is synchronized with
the current action.
*
Added filter export to VBScript and JScript - this can be used to very easily
create code for ASP or web pages or simple scripts.
*
Added COM interface to create Head/Tail filter, Run External Program filter,
Split filter, Show/Remove Duplicates filter, Secondary Output filter. Logging
functions, Input filter and Output filter settings can now be controlled with
precision.
*
COM functions are now error trapped to prevent dialog boxes appearing in
server versions. Error messages are captured to the ErrorText property.
*
The Script Filter's default script can now be set.
*
Fixed display issue where memo field containing multi-line text (such as the
search and replace fields) scrolled to the bottom rather than displaying text
from the top.
*
The command line text of the External Program filter now does not have double
quotes removed from it.
*
TextPipe now can paste more than 64K of text at a time into a memo control.
Previously controls could hold more than 64K of text, you just couldn't paste
that amount.
*
TextPipe now forces text paste operations to be plain text.
*
Speed improvements with string translation caching.
*
Wizard form now aligns correctly.

What's New in TextPipe v6.3.6
=============================

*
A new option to only output modified files was added to the Output filter.
*
A new Search option on the File Menu allows filters to be easily located.
*
The status window now shows how many bytes of the input file have been
processed.
*
A quick short cut is now to double click on any of the filename fields to
simply open the file (e.g. Single file output).
*
The Remove and Remove All buttons now work for all tabs (previously they were
not used for the Comments, Scratch Pad and Trial Run tabs).
*
When filters are being run from the command line, a new blank filter window
is no longer created - this means that the status of the already running
filter can be easily seen.
*
The Execute Program filter now deletes the Input and Output files when
processing has completed. This filter is commonly used for sorting very large
data files and other external processing.
*
The context grep filters Remove matching/non-matching lines and Retain
matching/non-matching lines have been re-designed so that they work exactly
the same as the Select matching/non-matching lines filters.
*
The /I command line option (recurse subfolders) now also applies to file
lists loaded using /L=<filename>.
*
Added new command line option /CLEARFILES to clear the file list. Also added
/S0-/S6 to handle different search/replace types.
*
Fixed bug with Remove/Retain Lines Matching Perl Pattern List, where all
backslashes needed to be doubled twice.
*
Fixed bug with none of the /C... command line parameters working.

What's New in TextPipe v6.3.5
=============================

*
Fixed problem with Extract Email Addresses filter removing one extra
character from the start of the email address when it started with mailto:
sendto:, email= etc.
*
Improved window behaviour when TextPipe is activated from COM - the main
window is now not visible unless it is specifically shown.
*
Improved COM handling - TextPipe exits correctly now that there are no
internal references to the Application object.

What's New in TextPipe v6.3.4
=============================

*
Fixed problem with saving Count Duplicate Lines and Remove Duplicate Lines
filters.

What's New in TextPipe v6.3.3
=============================

*
Fixed problem with Remove Patterns From List filter - was crashing.

What's New in TextPipe v6.3.2
=============================

*
Fixed problem with multiple output filters overwriting the same file.

What's New in TextPipe v6.3.1
=============================

*
Fixed sub filters not working for filters Add Text Side by Side, Add
Repeating Text Side by Side, Remove Lines Found In, Retain Lines Found In,
Remove Matching Lines Found In, Retain Matching Lines Found In.
*
Added new options to the Export Command Line of the Output filter.
*
Fixed problem with memos not displaying text properly.

What's New in TextPipe v6.3
===========================

*
Added new Find in HTML helper window to help users Edit, Remove, Replace or
Append to text inside HTML tags (such as the ALT tag of IMAGES), between HTML
tags (such as between <TITLE> and </TITLE>), and between two arbitrary pieces
of text.
*
Added new 'Select each line in turn' filter. This is a short cut for creating
a 'Select matching lines' filter with a pattern of '^'.
*
Window Menu now works properly - workaround found for Windows bug (being to
turn off AutoHotKeys and AutoLineReduction).
*
New System menu item (on task bar) to force TextPipe to close, even if it has
been activated via COM or from DataPipe etc.
*
Fixed double quotes appearing in filename fields after a filename has been
chosen.
*
When special characters are pasted, the default mode has been changed from
decimal to hex (so where \018 would previously have been pasted for special
character 18, now \x12 would be pasted)
*
New Word Wrap toggle item in the popup menu of memo fields.
*
New Start Column and Length options for Sort filter, Remove Duplicates filter
and Count Duplicates filter.
*
When a command line filter has been started with /G, TextPipe no longer
clears the filter list so that you can now see the contents of the command
line filter.
*
New command line parameter /OUTPUTKEEPSTRUCT to control whether the input
file's folder structure is retained when a new output folder is set.
*
Fixed a bug with dragging and dropping filters, where you could get an Access
Violation trying to edit the filter after moving it from inside its parent
filter.
*
Fixed @inputFilename and @fullInputFilename incorrectly using the output
filename.

What's New in TextPipe v6.2.9
=============================

*
Expanded the Special\Find and Replace menu item into an entire menu to make
it clearer.
*
Now uses Windows standard dialog boxes for easier translation.
*
Removed status window control resizing.
*
Removed validation on perl patterns that prevented back references from
working.
*
Fixed error with perl ^ not matching the start of the first line in files
less than 4K.
*
Fixed error with Extract Filter - some non-matching text was being passed to
the output.
*
Fixed the maximum allowed value on various filters - Split At Lines, Detail
lines allowed before writing to log file, size of random input file that can
be generated, insert column position, line numbering, context grep lines
before and after, delete columns values, duplicates filter, record length on
EOL filter, head/tail filter, all number-type filters like spaces to tabs,
centering etc, maximum buffer size for pattern matching etc.
*
Fixed handling of single and double quotes inside database queries.
*
Database filter 'insert script' option was not modifying file changed status.
*
Fixed TextPipe.getGlobalVar - now returns a blank string if the variable has
not been defined, instead of the variable name preceded by '@'.
GlobalVariables are now case-insensitive, and can hold multi-line values.
*
Fixed bug with 'Remove Prompting' not setting the modified flag if the filter
changed.
*
Changed the text qualifier for database insert scripts from " to '

What's New in TextPipe v6.2.8
=============================

*
All fonts are now set to the system default font. This makes translation of
Eastern languages far easier.
*
Tools\Put Sample In Trial Input now only includes files that have the 'Add'
action enabled. It also handles the situation where no files have been
specified.
*
Better DataPipe integration - now no longer hides the window when TextPipe is
closed. The main form minimizes instead.

What's New in TextPipe v6.2.7
=============================

*
Improved Command Line Wizard dialog.
*
New right click menu function for search/replace filters "Allow RTF tags"
converts the pattern to one that allows embedded RTF font tags.
*
Better integration with DataPipe - COM functions have changed, now
Application.hide minimizes TextPipe, and Application.show restores it.

What's New in TextPipe v6.2.6
=============================

*
Improved speed of COM functions.
*
Added COM code to support DataPipe
*
Fixed bug (again) where null lines caused the pattern matcher to fail.

What's New in TextPipe v6.2.5
=============================

*
Race condition fixed where a command line processing thread started with /G
/Q doesn't get a chance to start before the application exits.
*
New command line parameters /MAXIMIZED and /HIDDEN (hidden acts the same as
/MINIMIZED)
*
COM functions execute() and executeClipboard() now don't return until the
processing thread has completed.
*
New COM function isRunning to determine if a processing thread is still
running.
*
The COM function addReplaceFilter()'s ignoreCase parameter has been renamed
to matchCase - now the sense of the value is correct, and is consistent with
the user interface.
*
Fixed bug with perl-style patterns when used with look-behind assertions -
100 characters of text preceding the match is now retained in the search
buffer.

What's New in TextPipe v6.2.4
=============================

*
Secondary Output Filters can now be disabled and deleted.
*
Fixed errors occurring when a filter was modified after it had been moved up
or down using the up or down buttons.
*
Added Application.Quit function to COM
*
When created from COM, the default is now for the main form to be invisible.

What's New in TextPipe v6.2.3
=============================

*
Secondary Output Filters can now be added throughout the filter list to
duplicate the output file, or write selected lines and columns out to a
separate file, eg to place error lines or records in an error file, or to
separate records of different type or length into new files.
*
A new Select Block filter allows undelimited blocks of data to be selected
and processed as one.
*
Fixed VB/Jscript timeout - value is in milliseconds, not seconds.
*
Fixed obscure Perl replacement bug when run from command line.
*
Validation of ActiveX scripts now only occurs for VBScript - JScript
validation didn't work because the MS JScript engine does not implement
various functions.

What's New in TextPipe v6.2.2
=============================

*
Updated perl library to PCRE 3.7 (from 3.5)
*
EOL filter now allows more than one character in the replacement string.
*
Enabled some command line operations to provide Windows Explorer support in
TextPipe Web and Standard.
*
Fixed problem with the About dialog (in the evaluation version) interfering
with filter loading when TextPipe was invoked by double-clicking or opening a
.fll file.
*
Fixed bug with 'Escape Special Characters' not working properly.
*
Fixed bug with Split After XX Lines filter only outputting one line.
*
Fixed bug with 'Include Lines from End of File' when the number of lines was
greater than 255.

What's New in TextPipe v6.2.1
=============================

*
Tree view now displays more than just the first line of scripts and comments.
*
Fixed bug with perl-style grep and blank lines.
*
Input Filter export showed incorrect value for Delete Input Files After
Processing.
*
Filters loaded from the command line now have their full path saved, so they
get saved to the correct location even if the current directory is changed.
*
Added missing properties to Filter Export function.
*
Fixed bug handling command line filenames with embedded '..' paths.

What's New in TextPipe v6.2
===========================

Note: v6.2 was released quickly to correct some serious bugs, but it was not announced
until v6.2.1 was released with further bug fixes.

*
New filter to Remove lines matching any patterns in a list.
*
New filter to Retain lines matching any patterns in a list.
*
New filters to Retain matching lines, Remove matching lines, Retain non-
matching lines and Remove non-matching lines - this is easier to understand
than the Select lines filters.
*
New filter to add a file side-by-side with an existing file.
*
New filter to add a file side-by-side with an existing file and repeat the
added file if the file being added to has more lines.
*
New filter to Split a File after a given number of lines.
*
New Input Filter to generate random a data file of a particular size.
*
New filter to repeat a file's contents a specified number of times.
*
Added Script Timeout option to VBScript/JScript filter
*
Show Command Line dialog now also shows the current path and all environment
variables.
*
Added 'Escape special characters' menu item to right-click menus of all
fields. This is very handy for adding backslashes in front of pattern
matching meta characters, and doubling '@' and '$' characters where
appropriate.
*
Added 'Allow Extra Spaces' menu item to right-click menu of all pattern
fields, allowing you to easily search and replace in HTML files by ignoring
any extra spaces or tabs that commonly occur.
*
Added 'Allow Extra Line Feeds or Spaces' menu item to right-click menu of all
pattern fields, allowing you to easily search and replace in HTML files by
ignoring any extra spaces, tabs or line feeds that commonly occur.
*
Updated help, fixed application links to help file, help file now displays
when TextPipe is first run
*
Fixed bug with filter loading
*
Made splitter in Search/Replace filter visible, now resizes correctly.
*
Fixed 'Add folder' button - newer versions of Windows caused this dialog to
appear behind the main window.
*
Unknown field codes (a name preceded by an '@' symbol) are now NOT replaced
with a blank - they are left as is to prevent inadvertent use of a field code
causing text to disappear. Known field codes (@clipboard, @inputFilename,
@fullInputFilename) still work as normal.
*
Added new COM function to filter windows - closeWithoutSave. Added new
options to COM interface, including default values.
*
Fixed bug with a filter validation function not being called when there were
sub filters anywhere above them in the filter list.
*
Fixed a bug affecting the Select Matching Lines, Select Non-matching Lines,
Select Lines From Length To END, Select Lines From Start To End, Select Lines
From Start For Length filters when immediately followed by a line-based
filter such as a Reverse Each Line, Pad To Width etc. Some text was not
output at all.
*
Fixed a bug affecting the Remove Lines Found In List and Only Keep Lines
Found In List filters - because the list was not internally sorted, the
actual lines would only occasionally be processed correctly. A workaround was
to ensure that the line list is first sorted in ascending alphabetical order.

What's New in TextPipe v6.1.1
=============================

*
Improved the email address extraction filter to automatically remove leading
'mailto:', 'mime:', 'smtp:', 'email=', and remove trailing '?Subject' and
trailing periods.
*
Fixed Perl Pattern Match not saving 'greedy' setting
*
Fixed hidden form problem with active field being invisible
*
Fixed Access Violations on editing filter list

What's New in TextPipe v6.1
===========================

*
PERL-STYLE pattern matching! - use both greedy and non-greedy matching
options, as well as embedded comments, assertions, non-capturing sub-
expressions and more!
*
Brief-style pattern matching
*
MS Word style pattern matching - use the *, ?, etc wildcards that you can use
in MS Word.
*
New 'Extract' option for search and replaces, all non-matching text is
discarded.
*
The Found Text can now be referred to in the Replacement Text as '$0' for
Exact, Sounds-like and Edit-distance matches. Previously the Found Text could
not be referred to, making some replacements harder.
*
You can now easily add a sub filter to the selected filter by holding [Alt]
down while choosing it from the menu or selecting its speed button.
*
You can now easily add a filter at the parent level of a sub filter by
holding [Shift] down while choosing it from the menu or selecting its speed
button.
*
New Input Filter setting for text to come from the Trial Run Input area.
*
The filters Add Left Margin, Add Right Margin, Add Header and Add Footer now
do not parse the included file for special functions or characters. This
ensures that the file being included is included exactly as is. The parsing
of text specified in the edit field continues as previously.
*
Fixed a recursion bug when [Alt+Down] is pressed to move a filter and its sub
filters down, now the filters are moved down correctly.
*
Fixed bug with base panel of search and replace filter disappearing after
returning from the Trial Run area.
*
Removed the Font Control context menu item for controls.
*
Better handling of Cancel button during a processing run.
*
Fixed handling of NULL values with database filter and Generate Insert Script
mode.
*
Database Command Timeout setting for queries that run for more than 30
seconds.
*
Improved Tree handling is quicker and easier to use; the treeview changes
immediately when a setting is changed, and there is a new Up and Down button
to easily move filters up and down (in addition to Alt+Up and Alt+down).
*
All command line defaults now come from the Normal Template normal.fll, not
the internal defaults which used to be:
  inputFromFiles, promptOnEach=false, skip binary files,
binarySampleSize=100, outputToFiles, testMode=false, retainDateTime=false,
changeExtension=false, changeBaseDir=false.
*
Fix for the error " The Decision Cube Capacity is Low. Please deactivate
dimensions or change the data set"

What's New in TextPipe v5.6.3
=============================

*
Uninstall now removes all registry entries
*
Installer now supports unattended installs
*
Fixed short cut conflict between Remove/Remove All buttons and menu items
*
Fixed bug with registry security in registered version

What's New in TextPipe v5.6.2
=============================

*
Added 'Normal Template' functionality - a filter that is always loaded to set
defaults.
*
Delete Input Files after Processing setting now displays a warning box when
it is enabled. When a new filter is created this setting always defaults to
disabled.
*
Added shortcuts to tabs on status form, query replace form and command line
display form.
*
Fixed error in COM documentation for AddFile function - constants should be
1,2 and 3, not 0, 1 and 2.
*
Changed sizes of GUI controls to show foreign languages better.
*
Fixed bug with loading of foreign language, and enabled support for radio
group type controls.
*
Fixed bug with Append Mode.

What's New in TextPipe v5.6.1
=============================

*
Clicking on the filename column in the file list now sorts the filenames.
*
Improved COM interface - new methods for compiling the filter list which
improves speed if the same filter is used repeatedly
*
New Append to existing file setting

What's New in TextPipe v5.6
===========================

*
Localisation complete - now everything in TextPipe can be translated to a
foreign language by translating the file Languages\English.ini

For more details on how to translate TextPipe, please visit:
  http://www.datamystic.com/textpipe/translate.html
*
COM interface now allows TextPipe to be fully controlled from languages such
as VisualBasic, VBScript, C++, Delphi etc

For more details on how to translate TextPipe, please visit:
  http://www.datamystic.com/textpipe/com_interface.html
*
Fixed bug with dragging and dropping filters onto a filter that doesn't
support subfilters - now adds the dragged filter above it

What's New in TextPipe v5.5.2
=============================

*
Added code for TextPipe Pro Run Time Engine
*
Marketing changes for evaluation edition
*
Began preparation for localisation
*
Fixed command line processing bug with modifying the Output filter settings
after loading a filter with the /F= command
*
Fixed bug with debug windows

What's New in TextPipe v5.5.1
=============================

*
Forms and menus can now be translated into other languages via a .ini file in
the Languages folder
*
Improved the load time of massive filter lists loaded via the /Z= command
line parameter
*
Trial Output now shows Column and Row in the same way that Trial Input does
*
File size for files larger than 2GB is now displayed correctly
*
Improved the way that form sizes and positions are saved
*
Improved the way that registered user data is saved under Win2K
*
Forced installation to require the Administrator account to prevent registry
writing problems under Win2K

What's New in TextPipe v5.5
===========================

*
Lines of context surrounding a matching line can now be displayed, both
before and after matching or non matching lines (grep filter)
*
Added short cuts to tabs (Alt+1 through Alt+5)
*
Added Filter Wizard to help new users
*
Added command line options for setting the output file extension, output
folder, and merge output filename.
*
Sub filters can now be created by holding down Shift as they are drag and
dropped to a location inside another filter
*
The database filter now directly supports creating a database insert script
from extracted data
*
Improved tutorial lessons
*
TextPipe tutorial now listed in the start menu
*
Fixed bug with default folder not being TextPipe's folder (default was My
Documents folder)
*
Fixed bug with access violation on exit after a filter is invoked from the
command line
*
Fixed bug with Count Duplicates filter missing some lines
*
Fixed bug with Extract Emails filter missing the last email address if it
ended at the end of the file
*
Fixed bug with errors not being reported during trial run - now if errors are
reported the status dialog will not automatically close
*
Changed default extension to .fll for link filter dialog. Also added an open
button so that the filter can be easily loaded and edited

What's New in TextPipe v5.4.1
=============================

*
Can now CHANGE THE REPLACEMENT TEXT during a run. This is very useful for
fixing up mistakes in the replacement text, and can also be used for pattern
matches
*
New Paste Special command on context menus to allow pasting of special
characters without TextPipe converting them to its internal code
*
Unknown '\' entries are now output with the leading '\' rather than without.
This is to help new users
*
Trial Input and Output no longer attempt to validate special characters such
as \t etc.
*
Fixed bug with '\' being incorrectly interpreted in the Prompt for replace
window
*
Fixed bug with filters becoming modified when Search/Replace filter was
selected (not edited)

What's New in TextPipe v5.4
===========================

*
VBScript/JScript code can now control TextPipe - LogError, LogInfo,
inputFilename, fullInputFilename, terminate, setGlobalVar, getGlobalVar,
subFilterEntireText, version
*
VBScript/JScript code can now have subfilters. Subfilters can be turned on or
off by the script at any time using InitSubFilters, FlushSubFilters,
subFilterChunk
*
Find/Find Again in all memo fields (multi-line edit)
*
Font control dialog on all edit fields (including memos)
*
Extended the number of filters than can be invoked via the command line
*
Facility to generate TextPipe command line parameters from an existing filter
list
*
Trial Run can now be interrupted - runs in separate thread
*
Ctrl+A (Select All) now works in all fields
*
More options in the Window menu
*
Adhoc Transform tab renamed to Trial Run tab
*
Task bar menu now sends TextPipe to the back after a menu item is selected
*
Better handling of maximized windows when TextPipe restarts
*
Fixed bug with Select lines filter - if the input had (say) 2 lines and the
setting was from line 3 to END

What's New in TextPipe v5.3.9 Beta
==================================

*
Added drag and drop between filter windows
*
Added drag and drop promotion/demotion and movement of filters in the filter
tree display (and removed the Up/Down buttons)
*
Added Lesson 4 to tutorial

What's New in TextPipe v5.3.8
=============================

*
"x2" function now duplicates sub filters at the same time
*
Fixed bug with Insert Column At filter where the column was set to zero (ie
insert at end of line) and the actual line length = 1 (the text was inserted
at the beginning of the line)
*
Fixed bug with replace filter and whole words option - output text can be
duplicated
*
Fixed bug with merge filter and single file output filter
*
Fixed bug with Close All function not closing all windows
*
Fixed bug with Insert Filter File function changing the name of the currently
loaded filter

What's New in TextPipe v5.3.7
=============================

*
Updated Export To Clipboard and Print facility to cope with new filters and
tree structured filters. This facility is an excellent way to maintain
version information about your filters
*
Maps now moved to separate 'maps' folder
*
textpipe.log is now created in the same folder as textpipe.exe if no path has
been specified
*
Recent Files menu now uses submenu of main menu
*
Filters opened will now load into the New Filter window if it is unchanged
and does not hold a filter
*
Fixed problem with files and folders added via the command line not
defaulting the subfolders setting correctly
*
Fixed problem with filter opening twice when opened from Explorer when
TextPipe isn't currently opened
*
Fixed bug with merge filter not performing any filters following it in the
filter list

What's New in TextPipe v5.3.6
=============================

*
Added ability for Windows Explorer to detect TextPipe and open a filter in an
existing TextPipe window
*
Recycle bin functionality temporarily removed while bugs fixed

What's New in TextPipe v5.3.5
=============================

*
Fixed buttons and help for Files Tab
*
Delete file operations now use the Recycle Bin
*
Fixed bug with Remove Lines From File and Only Show Lines From file filters

What's New in TextPipe v5.3.4
=============================

*
(Hopefully) fixed long-standing error on exit

What's New in TextPipe v5.3.3
=============================

*
New subscribe facility on help menu
*
Fixed regexp bug with characters sets [] involving ranges of characters
specified with a \ eg [\000-\255]. Also error message for character sets
containing a \000.

What's New in TextPipe v5.3.2
=============================

*
Better error handling when file delete/rename fails
*
Fixed bug with number of subfolders to process (1 Level was same as None)
*
File\Open now always opens a filter in a new window. New menu item to open in
the current window
*
Fixed bug with Remove button, only removed first file from file list even if
multiple files were selected
*
Now checks if log file exists before using Notepad to open it
*
Double quoting of filenames and folders containing spaces now handled
correctly
*
Better handling of dropped files

What's New in TextPipe v5.3.1
=============================

*
Added option to delete input files once they are processed - to support
integration with GFI FAXmaker

What's New in TextPipe v5.3
===========================

*
New Sorting options
* ASCII Sort (case-sensitive and case-insensitive). Previously only an ANSI
sort was available (where lower case letters sort before upper case letters)
* Numeric sort
* Sort by length of line
*
New Unicode Map filter for translating Unicode files
*
Improved online help with Quick Start guide, improved tutorial and more
cross-referencing
*
Status form now has indicator to show whether test mode or overwrite mode is
active
*
All memo fields now can save to and load from a file
*
Trial output now has warning message if the output text contains null
characters
*
Improved Command Line Assistant dialog
*
Fixed bug with Query Replace window not scrolling found text into view
*
Fixed bug with "First Match Only" option discarding text
*
Fixed move filter up/down bug with filter lists containing more than 256
filters
*
Fixed bug with database filter not extracting data correctly when fixed width
format with no column titles was performed
*
Database filter now correctly handles non-SELECT queries that return a record
set
*
Fixed bug with access violation on exit
*
Fixed bug with pattern match ignoring First Match Only setting under some
circumstances

What's New in TextPipe v5.2
===========================

*
Added help toolbar for easier access to helpful features
*
Improved drag & drop behaviour for .fll files
*
Added automation check to disable debug filters
*
Added better error handling for MIME decode quoted printable filter
*
Advanced logging details in results window
*
Added error checking to ensure file can be deleted/moved to correct final
filename, or error logged if not
*
Added output option for all files merged to a single file
*
New Clear All and Paste context menu item
*
Prompting dialog for Read Only files, Prompt Before Processing and Prompt on
Binary Files now has option for No To All and Yes To All
*
New Search/Replace option to only replace the first occurrence
*
Trial Input/Output now have horizontal scroll bars
*
Grep filters (Keep matching lines, remove matching lines) now as select
filters, where matching lines get subfilters applied to them
*
ActiveX scripting no longer requires dummy parameters for functions
*
Improved match information tab for Edit Distance and Sounds-like matches
*
Context menu on tree view for faster deletion/enable toggle
*
Database query now creates correct XML document. Handling of previously
unrecognised data types. Original query output as comment. Headers always
output even if 0 rows returned. Unique column names generated for all output
types. Encoding="UTF-8" removed from XML header - not supported by some
versions of IE
*
Status dialog now shows count of Non-binary Files Skipped
*
New Edit Distance option for search/replace filters - very handy for finding
and locating misspelt words
*
New option to retain the existing file list when opening a new filter
*
New facility for search/replace to replace all or none within the current
selection
*
Loading a .fll filter file now opens in the same window
*
Better detection of filter list changed
*
Better handling of loading very old filter lists
*
Added force lines to width filter
*
Added advanced logging facilities with detailed tracking of all changes made
*
Fixed exception with exact match filter
*
Fixed handling of non-select statements in Database filter, and improved
error checking code
*
Fixed directory where preliminary output file created
*
Fixed split filter - was not performing "start file" processing for following
filters
*
Fixed Remove Email Headers - was crashing
*
Fixed bug that affects all line-processing filters - when using Mac files the
first character on each line may be repeated
*
Fixed bug with exact match filter that caused text to be output multiple
times
*
Fixed Win NT bug with popup context menus on memo fields
*
Fixed delete columns filter
*
Fixed handling of grep/inverse grep filters
*
Comment filters now have vertical scroll bars for comment text
*
Fixed subfilter bugs where filter list was not built correctly, missing
several filters
*
Fixed bug with Select Lines filter not outputting its results
*
Fixed bug with Select CSV Fields filter
*
Fixed errors with start and end word for Edit Distance and Sounds-like
filters
*
Fixed split filter
*
Fixed handling of Select Columns with the 4th option
*
Fixed debug filter error
*
Fixed Select CSV/Tab field missing last field
*
Fixed handling of map menu and task bar menu

What's New in TextPipe v5.1
===========================

*
** Sub filters - drag and drop a filter onto another in the tree view and the
dropped filter will then process JUST the output from the parent node. Eg
drop a "Title Case" filter onto a "Search/Replace" filter to change the
search result to Title Case
*
** New Database Connection filter. Attach to any ADO/ODBC database and
execute any SQL command text against it. The result can be output as CSV,
Tab-delimited or XML (with automatically generated DTD schema). All
characters are appropriately escaped, including quotes within strings for
CSV, and XML special characters.
*
Pattern matching now extended to handle up to 36 ($0-9,a-z) bracketed
expressions. This is 26 more than any other product on the market.
*
New Select CSV Fields filter - select a group of consecutive CSV fields and
apply sub filters to the result by dragging and dropping them onto this
filter. Handles commas embedded in quotes etc
*
New Select Tab-delimited Fields filter - select a group of consecutive tab-
delimited fields and apply sub filters to the result by dragging and dropping
them onto this filter
*
New Select Line Range filter - select a group of consecutive lines and apply
sub filters to the result by dragging and dropping them onto this filter
*
New Select Column Range filter - select a group of consecutive columns and
apply sub filters to the result by dragging and dropping them onto this
filter
*
New Remove All filter - useful for subfilters
*
Convert EOL filter can now specify any string as the new EOL character
*
New Truncate to Width filter
*
File output has new "Maintain folder structure" option when outputting to a
new folder
*
Remove Columns filter now has more advanced options
*
New option Skip Prompt if Identical for Search/replace filters
*
ActiveX scripts can now easily output text at the start and end of each file
*
New tools added - option to verify a filter before it is run - performs
syntax check etc. Also added Windows calculator
*
Explorer extension - right click on files, folders or drives brings up
TextPipe options. New window on Tools menu to customise the Explorer
extensions
*
Command Line Assistant window - provides examples of how to run TextPipe from
the command line in a variety of modes
*
Show Command Line window - displays the command line parameters used to
invoke TextPipe - useful for debugging command line scripts
*
Check for TextPipe Updates window - check if you have the most recent
TextPipe by querying the DataMystic web site
*
TextPipe now interfaces to 2 external tools (see Help\Related Products menu)-
  JustData from www.justsoft.com.au
  Offline Explorer from www.metaproducts.com

GUI enhancements
*
Tree view for filters which allows multi-level filters to be collapsed
*
Multiple-document interface (MDI) allows multiple filters to be open at once.
New Window menu to support MDI interface
*
Dockable toolbars
*
Icons in menus
*
Right-click (context) menus on all edit/memo fields allow special characters,
common patterns, replace expressions and macros to be inserted, as well as
the normal clipboard operations
*
Can now set the font used in the Ad-hoc Input/Output area

Bug fixes
*
ActiveX script now works in non-ad-hoc mode
*
Trial Run area now does not interpret \n, \t etc
*
Split files at Character - fixed bug

What's New in TextPipe v4.6
===========================
*
Example code for the TextPipe Engine - a text transformation .dll that can be
downloaded from www.datamystic.com
*
New ActiveX scripting filter supports arbitrary user-written code in
VBScript, Jscript or PerlScript, PythonScript, REXXscript etc
*
Output To New Folder can now recreate multi-level folder trees and output to
a new drive
*
New Undo command to undo any changes to the current filter
*
New file grid button to Copy Row
*
New Extract filter to extract just the required text
*
Show Duplicates filter now called Count Duplicates, and outputs the number of
times found to the left of each item
*
New Tools menu item to pull sample data from the file list or clipboard into
the trial run area
*
New Tools menu item to move the Trial Run Output back to the Trial Run Input
*
Add Line Numbers filter now allows format of line numbers can be changed to
custom user-defined settings, including numbering in hexadecimal etc
*
Sort filter now has Ignore Case option
*
Header, footer, left margin or right margin can now insert either a file or
text. Cursor position now shown
*
Length of trial input and output area shown in both lines and characters.
Cursor position now shown
*
Warning indicator on status bar now shows if Test Run mode is active or not
*
Debug filter - creates a debug window showing the text passing through the
filter at that point
*
External filter files can now be linked to, providing true 'black box' style
re-use, using File\Link To Filter
*
Can now duplicate the current filter - this makes adding a large number of
similar filters very easy
*
Find and replace filters now use multi-line edit control - making it easier
to edit large selections. The new edit control also provides the length and
line count of the text
*
Maps now display a hex column in addition to the original Decimal and ASCII
columns
*
Changed handling of changing the replacement text for a pattern-based search
- now the default text is not modified for future replaces
*
Tips of the Day
*
Query replace window now supports resizing better
*
Changed registration URLs
*
The filter names shown in the filter list are now truncated, preventing
Windows crashes with very long filter names
*
Fixed logging functions which were going wild when more than 1000 files were
processed
*
Fixed pattern matching not handling \n \r \e \t \a
*
Fixed Randomize filter - was outputting blank text
*
Fixed pattern based search with patterns longer than 247 characters - was
truncating and matching early

What's New in TextPipe v4.5
===========================
*
Soundex search method to search for words that sound like another word.
Useful when you are replacing a typing error
*
New Preferences dialog
- Play sound on completion
- If no association, use user defined editor (need new options window)
*
Prepare filter for automation - removes all user prompting
*
Prepare filter for distribution - removes all test text
*
Can now insert a file with the Add Header, Add Footer, Add Left Margin and
Add Right Margin filters - browse button
*
Add Line Numbers now has option to skip blank lines, (ie omit count but keep
counting), as well as omit count and not count line)
*
Now can split files at a given character, after a specified number of
occurrences (and split before, after or on top of character)
*
ANSI to Unicode and Unicode to ANSI filters (single byte to double byte)
*
Comment filters now shown in italics, disabled filters shown with strike
through text, speed button to add comments
*
Better progress indicator - now much more accurate
*
Duration of filter job in seconds shown in status dialog
*
Increased size of edit fields for easier editing of multi-line text. Display
of text length and number of lines of all memo fields (like Header, Footer,
Left/Right Margin, Insert Column etc)
*
Performance improvement when handling very large files
*
Improved performance for Remove Blanks from Start of Line, Convert to Lower
Case, Convert to Upper Case, OEM to ANSI, ANSI to OEM
*
Query replace dialog now shows file currently being processed
*
Query replace dialog now shows filter information for both normal and pattern
based searches (so it is easy to see which text was matched by each
expression)
*
Improved command line automation support. Now can have comments embedded -
useful when converting Perl scripts to TextPipe scripts because code can be
commented out rather than edited out
*
Form resizing now handled better
*
Fixed bug with Query Replace when replacement text was modified before
replacing
*
Fixed bug with saving/loading of window positions
*
Fixed bug with detection of when the filter has been modified
*
Fixed bug with query replace 'Yes - file' button - did not change the first
entry

What's New in TextPipe v4.4
===========================
*
Name change from DropConvert to TextPipe
*
Regular expression search and replace - allows pattern matching (also known
as non-exact matches or regexps). Case-insensitive or case-sensitive matching
*
HTML Help integration
*
Prompt on replace. Can change the replacement text
*
Option to delete test files at end of test run
*
@inputFilename macro now allowed in filters for Add Text To Start Of Lines,
Add Text At End Of Lines, Add Column Of Text
*
New @clipboard macro to insert the clipboard contents
*
Can disable filters by double-clicking or pressing Ctrl+T
*
When either the error display or the file display is full it is sent to the
log file
*
/z command line parameter to read parameters from a file
*
Grep-based regular expression matching
*
In-place editing of file specifications
*
Inclusion and exclusion lists for file processing
*
File groups allow better handling of complex file groupings
*
Subfolder recursion can now be specified on a per-file basis. Improved
performance of subfolder recursion
*
Improved detection for binary files - sample size can be adjusted, and binary
characteristics are better identified
*
Case-sensitive replace - if found text is uppercase, replacement can be
forced to all uppercase etc
*
New Export Filter to clipboard feature. Useful for documenting filters in
Word
*
Enhanced detail when printing filters
*
File associations for .dCv test files
*
Status dialog now shows separate error tab, and shows files generated by each
filter, with the old and new size of each with a comparison display for quick
checks
*
Install/uninstall feature. Creates Start Menu items, desktop icon, Send To
menu and file associations for .fll and .map files and for text files
*
Scratch pad area for test text
*
Trial run area for iterative testing
*
Improved error log for command line initiation
*
Pasted text is scanned for binary characters - and they are converted to
readable text. Pasted text is now checked to see if it has already been
converted
*
Heaps of useful sample filters
*
Window sizes now saved and restored. Minimum window sizes enforced
*
File list now supports including and excluding files, file groups, subfolder
inclusion/exclusion on a per-line basis
*
Scratch Pad area for saving chunks of text
*
New Edit Menu with Cut, Copy, Paste and Disable/Enable Filter
*
Settings now saved on a per-user basis
*
Improved descriptions in filter window
*
Improved performance. Better use of memory (now uses far less)
*
Scrolling on drag of filters in filter list
*
More filters and options available from command line
*
Now handles filters from future versions better
*
Printer setup menu item now doesn't get lost :-)
*
Fixed bug with "Base map on filter list" function

What's New in TextPipe v4.3
===========================
*
Filter to insert a column of text
*
Filter to convert to ranDOm CasE
*
Filter to extract URLs. Very useful for retrieving mailto, http, https, ftp
and gopher addresses
*
Task bar menu. Allows filters to be loaded and run without leaving external
programs such as Word, enabling conversion to occur seamlessly without
leaving the currently running program
*
All status windows from completed tasks are closed when starting a new task
*
Input filter new options:
  Input from file or clipboard
*
Output filter new options:
  Output to file or clipboard (independent of Input filter)
  change file extension
  change base directory on output
*
Filter to only show duplicate lines
*
Filter to reverse each line
*
Filter to centre justify lines
*
Filter to right justify lines
*
Filter to remove lines contained in an external file
*
Filter to only show lines contained in an external file
*
Filter to run external program. Very useful for user-written filters or for
features not yet incorporated into DropConvert
*
Filter to add comments to the filter list
*
Character maths filter now shows hex and binary equivalents for operand
*
File history (up to 5 filters) shown in file menu
*
Tool bar buttons for commonly-used filters
*
Floating hints now used in preference to status bar
*
Explorer menus for .fll file type
*
Removed evaluation edition restriction of 5 filters
*
Now uses InstallShield install/uninstall

See http://www.datamystic.com for more details on TextPipe

What's New in TextPipe – pending
===========================================

*
64 bit Windows Explorer shell extension! – for quick launching of filters.
*
Simultaneous search now far faster and supports 100,000+ search phrases
(exact match only).