Hi,
Creating DataSources for File Source Systems
Use
Before you can transfer data from a file source system, the metadata (the file and field information) must be available in BI in the form of a DataSource.
Prerequisites
Note the following with regard to CSV files:
● Fields that are not filled in a CSV file are filled with a blank space if they are character fields and with a zero (0) if they are numerical fields.
● If separators are used inconsistently in a CSV file, the incorrect separator (which is not defined in the DataSource) is read as a character and both fields are merged into one field and may be shortened. Subsequent fields are no longer in the correct order.
Note the following with regard to CSV files and ASCII files:
● The conversion routine that are used determine whether you have to specify leading zeros. See Conversion Routines in BI Systems.
● For dates, you usually use the format YYYYMMDD, without internal separators. Depending on the conversion routine that is used, you can also use other formats.
Notes on Loading
When you load external data, you can load the data into BI from any workstation. For performance reasons, however, you should store the data on an application server and load it into BI from there. This means that you can also load the data in the background.
If you want to load a large amount of transaction data into BI from a flat file and you can specify the file type of the flat file, you should create the flat file as an ASCII file. From a performance point of view, loading data from an ASCII file is the most cost-effective method. Loading from a CSV file takes longer because in this case, the separator characters and escape characters have to be sent and interpreted. In some circumstances, generating an ASCII file may involve more effort.
Procedure
You are in the Data Warehousing Workbench in the DataSource tree.
...
1. Select the application components in which you want to create the DataSource and choose Create DataSource.
2. On the next screen, enter a technical name for the DataSource, select the type of DataSource and choose Copy.
The DataSource maintenance screen appears.
3. Go to the General tab page.
a. Enter descriptions for the DataSource (short, medium, long).
b. As required, specify whether the DataSource builds an initial non-cumulative and can return duplicate data records within a request.
Specify whether you want to generate the PSA for the DataSource in untyped format: If the PSA is not typed it is not generated in a typed structure but is generated with character-like fields of type CHAR only.
Use this option if conversion during loading causes problems, for example, because there is no appropriate conversion routine, or if the source cannot guarantee that data is loaded with the correct data type.
In this case, after you have activated the DataSource you can load data into the PSA and correct it there.
4. Go to the Extraction tab page.
a. Define the delta method for the DataSource.
b. Specify whether you want the DataSource to support direct access to data.
c. Real-time data acquisition is not supported for data transfer from files.
d. Select the adapter for the data transfer. You can load text files or binary files from your local work station or from the application server.
Text-type files only contain characters that can be displayed and read as text. CSV and ASCII files are examples of text files. For CSV files you have to specify a character that separates the individual field values. In BI you have to specify this separator character and an escape character which specifies this character as a component of the value, as required. After you have specified these characters you have to use them in the file. ASCII files contain data in a specified length. The defined field length in the file must be the same as the length of the assigned field in BI.
Binary files contain data in the form of Bytes. A file of this type can contain any type of Byte value, including Bytes that cannot be displayed or read as text. In this case, the field values in the file have to be the same as the internal format of the assigned field in BI.
Choose Properties if you want to display the general adapter properties.
e. Select the path to the file that you want to load or enter the name of the file directly, for example C:/Daten/US/Kosten97.csv.
You can also create a routine that determines the name of your file. If you do not create a routine to determine the name of the file, the system reads the file name directly from the File Name field.
f. Depending on the adapter and the file to be loaded, make further settings.
■ For binary files:
Specify the character record settings for the data that you want to transfer.
■ Text-type files:
Specify how many rows in your file are header rows and can therefore be ignored when the data is transferred.
Specify the character record settings for the data that you want to transfer.
For ASCII files:
If you are loading data from an ASCII file, the data is requested with a fixed data record length.
For CSV files:
If you are loading data from an Excel CSV file, specify the data separator and the escape character.
Specify the separator that your file uses to divide the fields in the Data Separator field.
If the data separator character is a part of the value, the file indicates this by enclosing the value in particular start and end characters. You enter these start and end characters in the Escape Characteristics field.
You chose the ; character as data separator. However, your file contains the value 12;45 for a field. If you set “ as the escape character, the value in the file must be 12;45“ so that 12;45 is loaded into BI. The complete value that you want to transfer has to be enclosed by the escape characters.
If the escape characters do not enclose the value but are used within the value, the system interprets the escape characters as a normal part of the value. If you have specified “ as the escape character, the value 12”45 is transferred as 12”45 and 12”45” is transferred as 12”45”.
In a text editor (for example, note pad) check the escape character and separator currently being used in the file. These depend on the country version of the file you used.
Note that if you do not specify an escape character, the space character is interpreted as the escape character. We recommend that you use a different character as the escape character.
If you select the Hex indicator, you can specify the data separator and the escape character in hexadecimal format. When you enter a character for the data separator and the escape character, these are displayed as hexadecimal code after the entries have been checked. A two character entry for a data separator or an escape sign is always interpreted as a hexadecimal entry.
g. Make the settings for the number format (thousand separator and character used to represent a decimal point), as required.
h. Make the settings for currency conversion, as required.
i. Make any further settings that are dependent on your selection, as required.
5. Go to the Proposal tab page.
This tab page is only relevant for CSV files. For files in different formats, define the field list on the Fields tab page.
Here you create a proposal for the field list of the DataSource based on the sample data from your CSV file.
a. Specify the number of data records that you want to load and choose Upload Sample Data.
The data is displayed in the upper area of the tab page in the format of your file.
The system displays the proposal for the field list in the lower area of the tab page.
b. Choose Copy to Field List to select the fields that you want to transfer to the field list for the DataSource. All fields are selected by default.
6. Go to the Fields tab page.
Here you edit the fields that you transferred to the field list of the DataSource from the Proposal tab page. If you did not transfer the field list from a proposal, you can define the fields of the DataSource here.
a. To define a field, choose Insert Row and specify a field name.
b. Under Transfer, specify the decision-relevant DataSource fields that you want to be available for extraction and transferred to BI.
c. Instead of generating a proposal for the field list, you can enter InfoObjects to define the fields of the DataSource. Under Template InfoObject, specify InfoObjects for the fields in BI. This allows you to transfer the technical properties of the InfoObjects into the DataSource field.
Entering InfoObjects here does not equate to assigning them to DataSource fields. Assignments are made in the transformation. When you define the transformation, the system proposes the InfoObjects you entered here as InfoObjects that you might want to assign to a field.
d. Change the data type of the field as required.
e. Specify the key fields of the DataSource.
These fields are generated as a secondary index in the PSA. This is important in ensuring good performance for data transfer process selections, in particular with semantic grouping.
f. Specify whether lowercase is supported.
g. Specify whether the source provides the data in the internal or external format.
h. If you choose the external format, ensure that the output length of the field (external length) is correct. Change the entries, as required.
i. If required, specify a conversion routine that converts data from an external format into an internal format.
j. Select the fields for which you want to be able to set selection criteria when you schedule a data request using an InfoPackage. Data for this type of field is transferred in accordance with the selection criteria specified in the InfoPackage.
k. Choose the selection options (such as EQ, BT) that you want to be available for selection in the InfoPackage.
l. If required, in the field type, specify whether the data to be selected is language-dependent or time-dependent.
7. Check, save and activate the DataSource.
8. Go to the Preview tab page.
If you select Read Preview Data, the number of data records you specified in your field selection is displayed in a preview.
This function allows you to check whether the data formats and data are correct.
Result
The DataSource is created and is visible in the Data Warehousing Workbench in the DataSource overview for the file source system in the application component. When you activate the DataSource, the system generates a PSA table and a transfer program.
You can now create an InfoPackage. You define the selections for the data request in the InfoPackage. The data can be loaded into the inbound layer of the BI system, the PSA. Alternatively you can access the data directly if the DataSource supports direct access and you have defined a VirtualProvider in the data flow.
Assign points if it helps
Regards,
Hareesh
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.