Strange behaviour of the FILTER operator

[ceedee666]

Hi ABAP experts,

I recently played around a little with table expressions. When using the FILTER operator I encountered an unexpected behaviour. Consider the following sample program:

REPORT zcd_test_filter.

TYPES: BEGIN OF grid_element,
         row   TYPE int4,
         col   TYPE int4,
         value TYPE string,
       END OF grid_element.

TYPES grid_type TYPE SORTED TABLE OF grid_element WITH UNIQUE KEY row col.

DATA(grid) = VALUE grid_type(
    ( row = 1 col = 1 value = 'Element 1,1')
    ( row = 1 col = 2 value = 'Element 1,2')
    ( row = 1 col = 3 value = 'Element 1,3')
    ( row = 2 col = 1 value = 'Element 2,1')
    ( row = 2 col = 2 value = 'Element 2,2')
    ( row = 2 col = 3 value = 'Element 2,3') ).

DATA(filtered_grid) = FILTER #( grid WHERE row <= 2 AND col <= 2 ).

cl_demo_output=>display( filtered_grid ).

I would have expected that the filtered_grid would contain the following elements:

    ( row = 1 col = 1 value = 'Element 1,1')
    ( row = 1 col = 2 value = 'Element 1,2')
    ( row = 2 col = 1 value = 'Element 2,1')
    ( row = 2 col = 2 value = 'Element 2,2')

However, when executing the program the filtered_grid only contains the following elements:

    ( row = 1 col = 1 value = 'Element 1,1')
    ( row = 1 col = 2 value = 'Element 1,2')

Can anyone explain this behaviour to me? I consider this result a bug. However, I might be missing something.

Thanks

Christian

[nomssi]

I can reproduce the bug in 7.50 SP02.

JNN

[ceedee666]

Forgot to mention this. I'm running this on 7.50 SP4.

Christian

[Sandra_Rossi]

At the first glance, it's very surprising, and the documentation doesn't give an explanation for this behavior, but after thinking, I guess it's normal and only a bug in the documentation.

I guess it's normal because FILTER does not support standard tables, which surprised me a lot at the beginning, so I think the ABAP guys did it on purpose so that the performance is always good (so, the first location is found using a binary search). It also means that the FILTER behavior should avoid a full table scan. Consequently, it stops at the first line which does not meet the criteria.

(PS: little bug in your question, the second code portion shows the 2 last lines with row = 1 instead of row = 2)

[ceedee666]

Hi Sandra,

could be.

However, if I only filter e.g. on ROW = 2 I get the 3 expected results. As soon as I add a second where clause the behaviour changes. I'm getting more and more sure this is a bug. Furthermore, in the current version FILTER is basically unusable as I'd always need to implement the functionality using LOOP in order to make sure I get all the lines I need.

Christian

[Sandra_Rossi]

According to my understanding, if you filter on ROW = 2 and COL <= 2 then it's normal to get the expected result, because the expected result lines are all consecutive (binary search on ROW = 2 followed by a loop up to the first non-matching line).

Of course, it's only a guess, let's wait for Horst answer 🙂

PS: another consequence (among probably many ones), if the first key was alpha, it shouldn't work for KEY CP '*A' as it would imply a full table scan. I guess <> (still on first key) can work as the performance can be kept high.

[nomssi]

Hello Sandra,

I agree with your analysis but I disagree with your conclusion: it is not normal and only a bug in the documentation. I expect a compile time error like in this case (error: missing part of the primary key)

FILTER #( grid WHERE col = 2 )

before calling it a bug in the documentation.

JNN

[Sandra_Rossi]

Agreed, if it works like I said, then it should be reported as a bug in the syntax checker.

[ChristianGünter]

I also think that this is a bug. The documentation says that FILTER is an optimized shorthand for the corresponding FOR expression. In this case I would write as follows.

DATA(filtered_grid) = VALUE grid_type( FOR line IN grid
                                       WHERE ( row <= 2 AND col <= 2 )
                                       ( line ) ).

And this expression creates the expected result.

Christian

[SuhaSaha]

Hello Christian,

As per the latest FILTER documentation,

A condition for the table key used in the FILTER expression must be specified after WHERE:

• In the case of a hash key, precisely one comparison expression c op f for each key component. The only relational operator allowed for op is =.
• In the case of a sorted key, an initial part of the key must be covered by comparison expressions c op f. op can be any binary relational operator.

Now if i take the phrase "initial part of the key must be covered" and cross reference it with the documentation on LOOP...WHERE optimization,

it seems like the initial part of the key can be covered only with the '=' operator.

BR,

Suhas

PS - I agree with JNN that the compiler should raise a "syntax error" as in case of LOOP...WHERE.

[Sandra_Rossi]

For LOOP AT, the "optimization of the WHERE condition" does not lead to a syntax error, it's just that the access is not optimized, i.e. there's a full access (from the start to the end of the internal table). It's only for FILTER that there should be a syntax error.

[SuhaSaha]

For LOOP AT, the "optimization of the WHERE condition" does not lead to a syntax error, it's just that the access is not optimized

Yeah true, thanks for correcting me. ATC and/or CI check throws a warning that the key access is not optimized and will result in a full table scan.

If i remember correctly, you'll get the syntax error for secondary keys.

[Sandra_Rossi]

Thanks for correcting my imprecision 🙂 yes, no syntax error for primary keys, and syntax error for secondary keys.

[horst_keller]

I tested in a 7.50 system and found the same wrong behavior.

ROW COL VALUE
1 1 Element 1,1
1 2 Element 1,2

I tested in 7.51 (SP01) and everything is fine.

ROW COL VALUE
1 1 Element 1,1
1 2 Element 1,2
2 1 Element 2,1
2 2 Element 2,2

A kernel correction is available with SAP Note 2306379, "Filter by value with the relational operators <= or >= does not work as expected". The correction goes back to Kernel 7.42 (ABAP 7.40, SP08).

[ceedee666]

Hi Horst,

thanks for the clarification. Until we get a 7.51 developer version I'll stick with the FOR variant Christian suggested then.

Christian

[Sandra_Rossi]

I tested in 7.40 SP 11 - kernel 742 SP 300 and found the same wrong behavior.

[p604431]

Hi Experts,

I think the bug is with less that equal sign.

DATA(filtered_grid)= FILTER#( grid WHERE row >=1 AND col >=2).

I tried the above line, it gave expected result in 7.40.

Regards,

Gokul