Allow for subselects

Hi,

I’ve come across an issue with subselects.

My first problem was that i couldn’t use the result of a _select in an equals comparison operator, later i found out i can use belongs for that. Now i don’t know if that will have performance penalties on some platforms, but i consider it for later concern.

First and foremost, i need to combine two select statements that are related.

Consider the following model:

db = DAL('sqlite:memory')
db.define_table('data',Field('effdt','datetime'),Field('v'), Field('s'))
db.data.insert(effdt=datetime.now(),v=10, s='mine')
db.data.insert(effdt=datetime.now()-timedelta(5),v=5,s='mine')
db.data.insert(effdt=datetime.now()+timedelta(5),v=15,s='mine')
db.data.insert(effdt=datetime.now(),v=10, s='other')
db.data.insert(effdt=datetime.now()-timedelta(5),v=5,s='other')
db.data.insert(effdt=datetime.now()+timedelta(5),v=15,s='other')

Now when i try to get the current v for both ‘mine’ and ‘other’ i use the following:

print('- #0 ----------')
print (db.executesql("""
select effdt, v, s
from data d
where effdt == (
  select max(d_ed.effdt)
  from data d_ed
  where d_ed.effdt <= $NOW
    and d_ed.s = d.s
)
""",dict(NOW=datetime.now())))

I’ve tried to move this into DAL code, and have been struggling with it for quite some time. Now i came up with this:

d = db.data.with_alias('d')
d_ed = db.data.with_alias('d_ed')
subselect = db((d_ed.effdt <= datetime.now()) & (d_ed.s == d.s))._select(d_ed.effdt.max())
result = db(d.effdt.belongs(subselect.replace(', data AS d',''))).select(d.effdt, d.v, d.s)

which fortunately produces accurate results. But i find it quite ugly to patch the subselect string with data that is quite ‘woven’ with the structure of the query. It’s not very ‘dynamic’.

I’ve tried alternate ways, like using an inner join on a query , but sqlite3 won’t allow me to subselect with multiple columns apparently so that was one was off.

So finally i came up with a little patch, shown below, to allow for an extra parameter to _select. Here’s a sample for it in use:

d = db.data.with_alias('d')
d_ed = db.data.with_alias('d_ed')
subselect = db((d_ed.effdt <= datetime.now()) & (d_ed.s == d.s))._select(d_ed.effdt.max(), outer_scoped=[d])
result = db(d.effdt.belongs(subselect)).select(d.effdt, d.v, d.s)

As you can see I added outer_scoped in which all tables and aliases are placed that need not be mentioned in the from clause in the sql statement, because they refer to some name scoped outside of this query.
Works like a charm.

I don’t have any experience with a pull request yet, but if you’d like i’ll give it a try.

Hoping for this addition to come through, cause it will save me a hell of a lot of hardcoded effective dated queries.

Thanks. PS.: I don’t want a workaround as proposed in the groups thread mentioned above. I have queries using tens of these constructs in one single sql statement. Hardcoding those disallows for later addition of web2py/pydal common fields to work without handcrafted maintenance and database independence.

Index: /pydal/adapters/base.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
--- /pydal/adapters/base.py (revision )
+++ /pydal/adapters/base.py (revision )
@@ -39,7 +39,7 @@
 SELECT_ARGS = set(
     ('orderby', 'groupby', 'limitby', 'required', 'cache', 'left', 'distinct',
      'having', 'join', 'for_update', 'processor', 'cacheable',
-     'orderby_on_limitby'))
+     'orderby_on_limitby', 'outer_scoped'))


 class AdapterMeta(type):
@@ -1092,11 +1092,12 @@
         args_get = attributes.get
         tablenames = tables(query)
         tablenames_for_common_filters = tablenames
+        outer_scoped = [t._tablename for t in args_get('outer_scoped',[])]
         for field in fields:
             for tablename in tables(field):
                 if not tablename in tablenames:
                     tablenames.append(tablename)
@@ -1161,6 +1162,8 @@
                         if not t in important_tablenames ]
         else:
             excluded = tablenames
+
+        tablenames = [t for t in tablenames if t not in outer_scoped]

         if use_common_filters(query):
             query = self.common_filter(query,tablenames_for_common_filters)

[gi0baro]

@remcoboerma you’re right, I’m sorry. Will include this on #332

[gi0baro]

@remcoboerma thank you! Also mind that it will be published only in the next release. If you want to use it in the meantime you should use the master.